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I . INTRODUCTION 


PROGRAM  PURPOSE  AND  GOALS 

This  program  has  been  oriented  towards  the  general  problems 
of  assessing  the  status,  potentials,  and  problems  of  operational 
manned  systems  and  of  dealing  analytically  with  that  variance 
in  system  behavior  attributable  to  its  human  members.  As  the 
core  problem  is  the  behavior  of  the  responsible  system  element. 
Command  and  Control,  the  specific  question  of  concern  is,  "How 
can  the  performance  and  effectiveness  of  operational  Command  and 
Control  (C&C)  be  better  measured,  analyzed,  and  evaluated?" 

The  purpose  of  the  program  has  been  to  resolve  these 
problems  through  the  development  of  methods  and  a methodological 
framework  for  dealing  with  C&C  as  an  integral  part  of  systems. 
(This  is  in  deliberate  contrast  to  the  seemingly  prevailing  view 
that  C&C  is  a separate  system,  solely  unto  itself.)  The  goal  of 
the  program  is  to  develop  an  applied  methodology  for  C&C  analysis 
and  evaluation.  Such  a methodology- is  one  consisting  of  concepts 
and  methods  organized  within  an  analysis  process  context  which 
will  provide  guidance  to  the  analyst  in  the  matters  of: 

a.  What  kinds  of  information  will  application  of  a concept 
or  method  provide? 

b.  How  can  the  method  be  tailored  so  as  to  more  efficiently 
provide  the  needed  information? 

c.  How  does  each  method  fit  within  the  overall  analytic 
process  spanning  the  interval  between  initial 
presentation  of  the  question  and  obtaining  the  final 
answer? 

d.  Given  the  value  of  the  information  to  be  gained  and  the 
resources  available  to  the  analyst,  what  methods  can 
best  be  combined  into  a total  C&C  analysis  and 
evaluation  program? 

PROGRAM  APPROACH  AND  METHOD 

Program  goals  have  been  approached  from  the  viewpoint  that 
the  analyst's  task  is  essentially  a problem-solving  one  involving 
question-answer  processes;  and  that  our  job  was  to  facilitate 
these  processes.  As  a result,  methodology  contents  consist  of 
foundational  concepts  which  will  aid  in  better  formulating  the 
questions  and  of  methods  which  will  aid  in  obtaining  better 
answers. 

The  study  methods  have  included  study  of  all  past  and 
current  schools  of  thought  felt  to  be  of  potential  value  and 
indepth  onsite  studies  of  operational  Navy  systems.  The  studied 
systems  have  been  complex  and  "rich"  ones,  primarily  ones  found 
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on  carrier  vessels  end  the  ASW  weapon  system  crews.  The  principal 
crews  studied  onboard  the  carriers  have  been  those  in  the  Carrier 
Air  Traffic  Control  Centers  (CATCCs'  and  the  reader  is  referred 
to  Finley,  et  al  (Ref.  7)  for  the  details  of  these  studies. 

The  advantages  of  these  methods  have  been  that#  on  the  one  hand, 
we  have  rather  fully  used  the  resource  of  current  scientific 
knowledge;  while,  on  the  other  hand,  this  use  has  been  tempered 
and  additional  inputs  have  been  made  based  on  what  is  viable  and 
valid  for  application  to  questions  regarding  the  tectl  world  of 
operational  Navy  systems. 

UNIQUE  ASPECTS  OF  THE  PROGRAM 

In  attempting  to  realize  goals  and  purposes  such  as  the 
foregoing,  it  has  been  necessary  to  look  at  some  frequently 
ignored  problems  and  to  take  some  unusual  approaches.  It  might 
be  of  interest  to  the  reader  to  review  some  of  the  more  unique 
aspects  of  this  program. 

PROBLEMS  Although  many  problems  received  a great  deal  of 
attention  in  this  program,  note  can  be  made  of  three  which  are 
special;  special  in  that  they  are  central,  critical,  and  often 
ignored  or  "skipped  over  lightly".  These  include:  (1)  How  to 

conceive  of  C&C  such  that  it  can  be  explicitly  dealt  with  as  an 
integral  part  of  and  the  responsible  element  in  any  system  that 
is  not  totally  automated;  (2)  How  to  conceptualize  the  system 
and  its  human  members  such  that  operational  system  performance 
and  effectiveness  variance  can  be  better  organized  and  accounted 
for;  and  (3)  How  to  realize  and  contend  with  the  costs  of 
analyses  in  terms  of  resources  available  and  information  gained. 
The  first  two  of  the  foregoing  really  add  up  to  one  basic 
concern:  How  can  the  C&C  element  and  human  members  of  a system 

be  related  analytically  to  - and  therefore  be  held  accountable 
for  - the  variances  in  the  performance  and  effectiveness  of 
operational  systems.  The  third  item  is  concerned  with  the 
relative  reasonableness,  efficiency,  and  efficacy  with  which 
alternative  analysis  and  evaluation  packages  can  be  performed. 

APPROACHES  Unique  aspects  of  the  approach  taken  include  the 
following: 

a.  Applications  Oriented  - the  above  are  real-world  problems 
encountered  when  dealing  with  real  operational  systems. 
The  areas  between  basic  and  applied  research,  and 
between  theory  and  field  operations,  are  the  relatively 
unexplored  ones  of  applications  research  and  applied 
methodologies.  It  is  these  areas  which  W(  ~e  felt  to 
provide  the  question-answer  relationships  needed  in  this 
program  in  order  to  make  general  scientific  knowledges 
applicable  to  specific  operational  problems. 

b.  Analysis  Process  Rooted  - to  develop  an  assortment  of 
concepts  and  methods  frr  a purpose  is  one  thing.  To 
organize  them  so  as  to  make  them  really  applicable  to  a 
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variety  of  problems,  i.e.,  to  develop  an  applied 
methodology,  is  something  else  again  and  requires  that 
they  be  developed  and  arranged  according  to  some 
organizing  principle.  The  organizing  principle  in  this 
study  has  been  the  analysis  process  flow.  Which,  of 
course,  presented  an  immediate  problem  in  that  this 
process  has  not  previously  been  greatly  discussed  or 
used  in  this  manner.  (Most  analysis  methods  books,  for 
example,  have  been  organized  according  to  either  theory 
or  problem  categories.)  It  was  therefore  necessary  that 
time  and  effort  be  devoted  just  to  the  definition  of 
this  process  (see  Chapter  II) . 


c.  Applied  Measurement  - there  is  a great  deal  of  general 
measurement  theory,  but  little  in  the  way  of  applied 
theory  or  method.  Each  measure,  with  its  associated 
measurement  procedures,  provides  data  containing  some 
particular  piece  of  information  - and  no  other.  It  is 
the  analyst's  task  to  select  those  measures  and/or  that 
already  available  measurement  data  which  will  provide 
him  with  that  specific  information  he  especially  needs. 
Although  the  mathematical  scaling  and  other  properties 
of  measure  are  indeed  important,  of  far  greater 
importance,  from  an  applied  question-answer  standpoint, 
is  the  definition,  or  meaning,  provided  by  data  on  that 
measure"  Concepts  and  methods  presented  in  the  sections 
that  follow  will  be  presented  from  the  viewpoint  that 

1...  ■ understood  meaning  is  the  goal  of  analysis  and  that 

valid  and  sufficient  measurement  - whether  qualitative 
(e.g.,  verbal)  or  quantitative  - is  the  means  to  that 
end. 

d.  Decision  and  Utility  Notions  - it  should  be  clear  by  now 
that  the  goal  of  this  report  is  not  to  enable  analysis 
for  analysis'  sake.  Rather,  it  is  to  enable  analyses 
regarding  C&  ' which  provide  relevant,  valuable,  and 
cost-effective  meaning  - i.e.,  information  which  answers 
the  questions  asked  within  the  value  and  resource  bounds 
set  for  the  analyses.  While  several  sections  address 
problems  associated  with  effective  management  of  analysis 
p -ams.  Chapters  IV  and  VI  are  the  most  directly 
coi.^erned  with  these  issues. 

A REVIEW  OF  PROGRAM  OUTPUTS 

Program  outputs  include  an  initial  study  progress  report, 
this  report,  which  is  the  centra)  and  final  report,  and  several 
supporting  documents.  So  as  to  provide  the  reader  with  a program 
overview  and  references,  the  initial  report  and  supporting 
documents  are  further  described  here. 

INITIAL  REPORT  A detailed  report  of  the  first  year's  study 
(.  ) methods,  progress,  and  findings  is  provided  by  the  following: 
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Finley,  D.L.,  Mtckler,  F.A.,  Gainer,  C.A.,  and  Roe,  W.T. 
Development  of  in  analysis  and  evaluation  methodology  for 
Command  and  Control;  First  technical  report.  Contract 
N00014-73-C-0095,  Nav.il  Analysis  Programs,  Office  of  Naval 
Research,  Arlington,  VA,  March  1974  (AD  778  028). 

First  year  results  included  definition  of  the  methodological 
framework,  the  identification  and  initiation  of  development 
effort  on  needed  concepts  and  methods*  and  the  provision  of 
recommendations  for  improvement  of  observed  fleet  systems  and 
operational  policies. 

SUPPORTING  DOCUMENTS  In  order  to  make  the  overall  methodological 
framework  more  evident  and  so  as  to  better  emphasize  the 
conceptual  network  and  contents  contained  therein  in  this,  the 
central  and  integrating  report,  some  materials  have  been 
presented  in  separate  reports.  These  are  materials  which  are 
essential  parts  of  the  methodology,  but,  nonetheless,  are 
sufficiently  developed  at  this  time  so  as  to  be  able  to  stand 
alone  as  individual  reports  as  well. 

The  reports,  in  order  of  production,  are: 

Roe,  W.T.  and  Finley,  D.L.  Ergonomic  models  of  human 
performance:  Source  materials  for  the  analyst.  Contract 

N 0 0 0 1 4- 7 4-C- 0324 , w ava 1 Analysis  Programs,  Office  of 
Naval  Research,  Arlington,  VA,  September  1975  (AD  ) . 

Obermayer,  R.W.  A computer  model  for  Command  and  Control 
analyses . Contract  N00C14--74-C-0324 , Naval  Analysis 
Programs,  Office  of  Naval  Research,  Arlington,  VA, 

November  1975  (AD  ) . 

Gainer,  C.A.  A handbook  of  systems  description  methods. 
Contract  N00014-74-C-0324,  Naval  Analysis  Programs,  Office 
of  Naval  Research,  Arlington,  VA,  December  1975  (AD  ). 

Finley,  D.L.  and  Muckier,  F.A.  Human  factors  research  and 
the  development  of  a manned  systems  applications  science ; 

The  systems  sampling  problem  and  a solution.  Contract 
N00014-74-C-0324,  Naval  Analysis  Programs,  Office  of 
Naval  Research,  Arlington,  VA,  December  1975  (AD  ) . 

These  documents  contain  source  materials,  methods,  guidelines  and 
procedures,  and  theory  background  and  development  details. 

A CLOSING  NOTE  A completely  developed  methodology  for  dealing 
analytically  with  questions  of  the  performance  and  effectiveness 
of  C&C  would  comprise  a much  heftier  central  volume  than  this 
and  require  several  more  supporting  documents.  Which  is  simply 
to  say  that  much  work  remains  to  be  done  on  this  topic  and  we 
hope  that  others  will  continue  where  we  have  left  off. 
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II.  DEFINITION  Ox  THE  ANALYSIS  PROCESS 


THE  ANALYST 

The  principal  user  of  the  C&C  analysis  and  evaluation 
methodology  is  considered  to  be  an  analyst  possessing  certain 
characteristics.  These  include: 

a.  Being  tasked  with  deriving  information,  principally 
through  measurement  and  analytic  procedures,  regarding 
C&C  performance  and  effectiveness  in  operating  systems. 
The  desired  information  is  that  which  is  evaluative, 
diagnostic,  or  predictive  in  nature  and  which  is  useful 
in  resolving  management,  planning,  and  resource 
allocation  problems. 

b.  Having  these  resources  available  to  him: 

(1)  The  normally  available  record  data,  which,  in  this 
report,  will  be  referred  to  as  "available  data" . 
These  include  not  only  that  data  obtainable  from 
computerized  data  banks,  but  also  that  which  is 
contained  in  files  and  on  tapes  of  much  lesser 
degrees  of  standardization  and  organization. 

(2)  The  ability  to  query  systems  personnel  so  as  to 
obtain  additional  qualitative  and  quantitative 
data. 

(3)  Some  data  retrieval  and  computational  capabilities. 

(4)  Some  ability  to  modify  normal  data  collection 
procedures  so  as  to  better  satisfy  current  or 
future  data  needs. 

c.  Being  required  to  constrain  the  use  of  the  above 
resources  within  reasonable  utility  and  feasibility 
bounds . 

The  user  of  this  methodology  is  also  assumed  to  be  a person 
faced  with  a question  which  requires  that  several  decisions  be 
made  in  the  process  of  obtaining  a final  answer.  The  methodology 
is,  as  can  be  seen  from  a review  of  the  Table  of  Contents, 
structured  to  a.i  •'  several  of  these  decisions.  An  underlying  set 
of  decisions  th  - analyst  is  considered  to  make  relates  to  the 
matter  of  what  constitutes  the  sequence  of  attribute,  performance, 
and  effectiveness  relationships  in  the  system.  That  is,  deciding 
what  are  the  parameters  of  and  relationships  between  (1)  the 
system  elements  and  components  at  any  one  level  of  definition, 

(2)  the  levels  of  system  definition,  and  (3)  the  systems  under 
study,  other  systems,  and  the  system  environments. 
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THE  PROCESS 

A basic  analytic  process  concept  was  developed  during  the 
first  year  of  program  effort  to  serve  as  the  organizing  principle 
for  n thodology  development  efforts.  The  graphic  representation 
that  was  used  at  that  time  is  repeated  in  Figure  1.  It  is  meant, 
by  its  layout,  to  convey  the  thoughts  that  (1)  analysis  should  be 
a sequential,  albeit  iterative,  process  and  (2)  tl  e steps  taken 
at  any  one  pc^nt  of  the  process  are  largely  determined  and 
limited  by  the  results  of  preceding  analyses. 

Since  then  the  concept  of  an  analysis  process  has  further 
evolved  such  that  a graphic  overview  would  now  appear  as  in 
Figure  2.  A more  detailed  listing  of  analysis  stages  is  the 
following : 

a.  Stating  the  question. 

b.  Taxonomization  of  the  universe  of  things  to  be  dealt 
with. 

c.  Initial  and  general  identification  of  measures. 

d.  System  identification  through  the  accumulation, 
analysis,  development,  and  preparation  of  system 
descriptive  information  and  the  "available  data". 

e.  Initial  specification  of  the  models  and  operations  and 
their  association  with  members  of  the  measures  set. 

f.  Final  specification  of  the  desired  members  of  the 
measures  set. 

g.  Final  evaluation  of  the  available  data  and  definition 
of  the  measures  represented  therein. 

h.  Final  evaluation  and  selection  of  the  measures  and 
measurement  data  samples  to  be  actually  used. 

i.  Final  formulations  of  the  system  models  and  model 
operations . 

j.  Final  selection  of  analytic  approaches  and  performance 
of  the  deterministic  and  stochastic  analyses. 

As  noted  at  the  bottom  of  Figure  2,  the  chapters  of  this  report 
support  the  identified  analysis  stages  and  have  been  organized 
accordingly . 

This  is  the  opportune  point  at  which  to  introduce  the  reader 
to  two  aspects  of  the  analysis  process  which  will  underly  the 
presentation  of  some  of  the  materials  in  later  chapters.  These 
two  aspects  are  presented  in  Table  1 and  Figure  3.  In  Table  1, 
the  possible  topical  components  that  might  be  contained  in  a 
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QUESTIONS  RE: 

COMMAND  & CONTROL  AND 
SYSTEMS  EFFECTIVENESS 


SYSTEMS  DEFINITIONS 
& OPERATIONS 
DESCRIPTIONS 


MEASURES  SPECIFICATION, 
ANALYSIS,  & EVALUATION 


DATA  ANALYSIS 


The  Basic  Analytic  Process  Concept  Developed 
During  the  First  Year  of  Program  Effort 
(taken  from  Finley,  et  al,  Ref.  7). 
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TABLE  1.  RELATIONSHIPS  BETWEEN  QUESTION  TOPIC  COMPONENTS  J 
THE  ANALYSIS  STAGES  OF  SYSTEM  DESCRIPTION,  MEASU 
DEFINITION,  AND  SYSTEM  MODELS  FORMULATION  (page 
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question  under  study  are  listed  and  related  to  three  major 
analysis  stages.  The  relationship  is  in  terms  of  impact  on  the 
analysis  steps  to  be  taken.  In  Figure  3,  effects  of  analysis 
process  stages  on  the  evolution  and  formulation  of  a system  model 
and  its  operations  are  depicted. 

Finally,  as  a closing  note  of  interest,  the  analysis  process 
stages  were  used  to  specify  the  sequence  of  steps  taken  in  two 
activities  of  concern  to  us  here:  (1)  the  design  and  execution, 

of  an  empirical  study  in  order  to  obtain  information  and  (2)  the 
design  and  execution  of  an  analytical  study  using  "available 
data"  for  the  same  purpose.  The  result  of  this  exercise  is 
presented  in  Table  2 and  it  is  interesting  to  note  how  similar 
the  processes  in  these  two  activities  can  be  made  to  appear. 
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TABLE  2.  A COMPARISON  OF  EMPIRICAL  EXPERIMENTS 

AND  DATA  BASE  ANALYSIS  PROGRAMS  (Page  1 of  2) 

PROGRAMS 


DATA  BASE  ANALYSIS 


EMPIRICAL  EXPERIMENTS 


PURPOSES 


CONSTRAINTS 


The  general  purposes  are 
to  collect  specially  de- 
fined data  which  can  be 
used  to  test  hypotheses, 
derive  parameter  value 
estimates,  and  derive 
function  definitions  - 
all  as  needed  to  answer 
a question 

Limitations  on  the 
amount  of  information 
potentially  contained 
in  "available  data" 
bases 


[Data  retrieval  capa- 
bilities and  resources 


Same 


Limitations  in  the  en- 
vironments, etc.,  avail- 
able for  study  purposes 
and  the  amount  and  nature 
of  situation  manipulation 
and  onsite  measurement 
possible 

Resources  for  and  adequacy 
of  manipulations  made  and 
measurements  taken 


Resources  available 
for  analysis 


Resources  available  for 
analysis 


Statement  of  question  Statement  of  question 


Step  b. 


Step  c. 


Step  d. 


Step  e. 


Step  f. 


Populations  identifica- 
tion and  definition 

Populations  sample 
specification 

Scenarios/conditions 

specification 

Specification  of 
measures  and  data 
samples 


Same  as  step  b for  data 
programs 

Same 


Same 


Same 


Analysis  and  evaluation  Analysis  and  evaluation  of 
"available"  data  re-  situations,  etc.,  avail- 
sources  able  for  experiment 

.conduct 


TABLE  2.  -continued-  (Page  2 of  2) 


Step  g. 


Step  h. 


Specification  of 
assumptions  and 
hypotheses 

Reiterate  Steps  b - e 


Same 


Same 


Step  i. 


Specification  and 
integration  of 
analytic  programs 


Specification  of 
experimental  design  and 
procedures 


Step  j . 


Step  k. 
Step  1. 


Specification  of 
procedures  for 
evaluation  of  analysis 
results 

Retrieve  data 

Reiterate  Step  j as 
necessary 


Specification  of  data 
reduction  and  analysis 
procedures 


Run  experiment 
Same 


Step  m. 


CONCEPTS  USED 
IN  THE 
PERFORMANCE 
OF  THE  AEOVE 
PROCESS 


paralyze  data 


Same 


Performance  of  above 
Steps  a through  h is 
guided  by  general 
models  - the  investiga- 
tor's views  of  systems, 
components,  elements, 
and  environments  with 
regard  to  their 
structure,  functioning, 
oehavior,  and  inter- 
actions - whether  or 
not  these  models  are 
explicitly  recognized 
and  acknowledged  by 
til-",  investigator.  The 
results  of  Step  i is 
the  formalization  of 
specific  cases  drawn 
from  these  general 
models . 
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COMMAND  AND  CONTROL:  DEFINITIONS,  DISCUSSIONS,  AND  MODELS 

DEFINITIONS  Because  there  are  many  conflicting  concepts  of  C&C, 
it  is  useful  to  state  here  what  definition  has  been  assumed  for 


this  report:  Command  and  control  is  the  management  component  of 

any  system. 


Some  clarification  and  expansion  of  this  definition  may  be 
useful : 


1.  C&C  is  a subsystem.  For  some  purpose,  it  may  be  useful 
to  consider  C&C  as  an  isolated  entity.  For  example,  when 
problems  have  been  identified  in  the  C&C  element,  those  problems 
may  be  examined  solely  within  the  element  itself.  But,  the  use 
of  the  phrase  "command  and  control  system"  is  a conventional 
convenience.  What  is  ultimately  of  interest  is  what  the  C&C 
component  does  in  relation  to  all  c '.er  components  of  the  total 
system. 


2.  C&C  functions  are  exercised  throughout  the  system,  not 
solely  at  the  "top"  of  the  system.  For  example,  directives 
issued  from  the  C&C  component  are  always  subject  to  some 

( interpretation  and  application  in  other  parts  of  the  system.  In 

the  act  of  applying  these  directives,  variations  - intended  or 
not  intended  - are  always  introduced.  In  every  real  system, 
unofficial  command  directives  and  actions  may  be  generated 
depending  upon  the  extent  and  degree  of  system  control. 

Flexibility  appears  to  be  essential  for  any  system,  and 
flexibility  implies  that  other  elements  of  the  system  have  some 
options  in  at  least  limited  execution  of  C&C  functions. 

In  the  vital  area  of  information  flow  through  real 
systems,  no  system  can  transmit  exactly  the  information  output  of 
the  C&C  component.  The  communication  message  may  be  the  same  to 
all  components,  but  the  interpretation  of  the  message  will  always 
vary.  Further,  information  supplied  to  the  C&C  component  has 
always  been  suspect  to  some  degree,  and  rightly  so.  Data  inputs 
to  the  C&C  component  must  be  filtered,  or  C&C  would  be  overwhelmed 
with  quantity  of  data.  But  the  act  of  filtering  inevitably 
distorts  the  data  being  transmitted;  the  C&C  component  must  test 
the  data  flow  process  to  insure  that  the  filtering  does  not 
change  meaning. 

3.  C&C  as  a subsystem  of  military  systems  is  equivalent  to 
civilian  management  systems.  Outside  of  the  element  of  personal 
risk,  there  are  no  significant  functional  differences  between 
military  and  civilian  management.  Differences  are  of  degree  and 

( ) not  of  kind. 
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Military  systems  are  often  characterized  as  examples  of 
strong  centralization.  Yet,  in  fact,  many  military  systems 
stress  at  least  temporary  decentralization  in  the  sense  of 
encouraging  individual  and  unit  initiative.  Many  civilian 
systems  have  greater  degrees  of  centralization  than  military 
systems. 


With  respect  of  physical  stressors,  military  systems 
represent,  of  course,  an  extreme  point  on  the  continuum  of 
management  systems.  But,  looking  at  psychological  and  social 
stresses,  civilian  management  systems  in  some  cases  create 
sustained  stress  conditions  continuously  for  periods  of  years. 

4.  C&C  is  not  a physical  location.  The  command  post  is 
not  the  C&C  component;  rather,  it  is  a tool  used,  or  not  used, 
by  the  C&C  component.  Producers  of  command  installations 
frequently  appear  to  confuse  structure  with  function. 

5.  C&C  is  not  solely  an  individual,  but  consists  of  all 
individuals  within  the  total  system  that  generate  and/or  execute 
C&C  functions.  It  is  convenient  to  identify  C&C  with  the 
commander,  but  in  fact  C&C  functions  are  distributed  through  many 
individuals  in  the  system. 

COMMAND  AND  CONTROL  FUNCTIONS  It  is  necessary  to  expand  the 
term  "management  component"  into  the  C&C  functions  performed  by 
the  command  component.  The  initial  technical  report  (Ref.  7) 
listed  six  major  functions  which  may  be  considered  a minimum  set 
of  functions  that  define  the  management  component.  These  are 
considered  to  be  necessary  although  they  are  probably  not 
sufficient.  In  summary: 

1,  The  C&C  component  establishes  general  and  specific  goals 
and  standards.  No  other  component  can  perform  this  function. 
Parenthetically,  this  does  not  necessarily  imply  a unilateral 
action  by  the  C&C  component.  The  goals  and  standards  may  be 
generated  jointly  by  such  methods  as  management  by  objectives. 
However,  this  concerns  the  methods  by  which  the  goals  and 
standards  are  derived  and  not  the  performance  of  the  function 
per  se. 


2.  The  C&C  component  establishes  procedures  and  techniques 
by  which  the  system  will  achieve  goals.  On  the  positive  side, 
this  assists  the  total  system  in  suggesting  ways  of  achieving 
goals.  Here,  the  desirable  degree  of  flexibility  is  an  important 
issue.  There  is  no  substantial  objective  evidence  that  any 
degree  of  specificity  is  better  than  any  other.  As  a heuristic 
in  practice,  it  is  probable  the  extremes  - total  or  no  specifica- 
tion - are  to  be  avoided.  It  may  be  that  the  optimal  level  of 
specificity  on  procedures  and  techniques  is  dictated  by  the 
mission  tasks. 
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3.  The  C&C  component  defines  the  constraints  under  which 
the  system  will  operate.  This  function  is  the  important  one  of 
'establishing  what  the  system  cannot  do.  Most  C&C  and  management 
systems  have  been  reluctant  to  perform  this  function  on  the 
grounds  that  it  may  restrict  the  system  in  performance.  Yet, 
many  system  problems  that  occur  constantly  could  be  avoided  if 
specific  constraints  were  stated  explicitly.  This  is  particular- 
ly true  of  the  utilization  of  the  personnel  component. 

4.  The  C&C  component  is  responsible  for  the  level  of  system 
performance  achieved.  Command  is  responsible,  and  command  is 
accountable, since  command  has  been  given  the  authority. 

5.  The  C&C  component  defines  the  nature  of  the  interaction 
between  management  and  the  rest  of  the  system.  This  function 
includes  both  organizational  structure  and  style  - at  least  to 
the  degree  that  such  processes  can  be  meaningfully  organized  by 
formal  action.  It  may  be  that  some  freedom  is  essential  in 
dynamic  organizations  for  these  parameters  to  develop  spontaneous- 
ly. C&C  can  then  codify  or  modify  the  interactions  as  necessary. 

6.  The  C&C  component  establishes  data  acquisition,  data 
processing  and  information  needs  for  all  levels  of  the  system. 

The  violation  of  this  function  has  led,  we  believe,  to  the 
current  state  of  ineffective  and  extremely  costly  communication 
within  systems.  Further,  as  will  be  discussed  later  (Chapter  VI), 
the  C&C  component  should  be  governed  by  a minimization  axiom  with 
respect  to  intra-system  communication.  Far  too  much  irrelevant 
data  are  being  exchanged  within  modern  day  systems.  This  seems 
particularly  true  if  the  system  has  abundant  ADP  capability. 

A MODEL  FOR  COMMAND  AND  CONTROL  The  previous  technical  report 
proposed  a generalized  model  for  C&C  as  shown  in  Figure  4 . This 
has  seemed  particularly  useful  in  maintaining  the  operational 
distinctions  between  command  flow  vs.  control  flow  vs.  data/ 
information  transmission  within  the  total  system  structure. 

Assuming  that  such  models  are  useful  (if,  indeed,  not 
essential)  to  better  understanding,  it  becomes  critical  to  be 
able  to  classify  system  structure  and  process.  Models  cannot  be 
built  without  classification,  and,  hence,  it  has  become  necessary 
to  explore  taxonomization  problems;  this  is  done  in  the  next 
two  sections.  Taxonomization  is  basically  concerned  with 
rational  description.  If  we  cannot  describe  a system,  it  is 
doubtful  that  we  can  understan 1 the  system. 
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TAXONOMIZATION  CONCEPTS 

Taxonomy,  or  the  science  of  classification,  is  concerned 
with  the  grouping  and  ordering  of  things  so  as  to  achieve 
meaning  and  manipulational  capability;  as  such  it  is  a most 
fundamental  instrument  in  the  development  of  a science.  In  the 
basic  sciences,  well-known  examples  of  taxonomies  include  the 
phyla  of  zoology  and  botany  and  the  periodic  table  of  chemistry. 
In  the  applied  science  of  human  factors,  we  have  descriptive  and 
analytic  task  taxonomies. 

In  this  business  of  grouping  and  ordering  collections  of 
things,  there  are  two  aspects  to  be  concerned  with;  one  is  the 
criteria,  or  rules  of  assignment  and  distinction,  used  to 
separate  members  of  a collection  into  their  respective  groups. 
That  is,  the  criteria  that  put  a taxonomy  into  operation.  This 
aspect  has  been  the  traditional  and  principal  concern  of  the 
science  of  classification  (cf,  ref.  19).  The  other  aspect  is 
that  of  taxonomization,  or  the  process  of  developing  the  taxonomy 
to  begin  with.  This” initial  process  stage  has  never  been 
examined  in  any  detail;  probably  because  it  is  a creative  act, 
an  act  requiring  talent,  and  as  such  has  been  assumed  to  be 
unexaminable . 

Perhaps  "...rushing  in  where  angels  fear  to  tread",  we 
are  not  only  going  to  look  into  the  matters  of  Taxonomy*  and 
taxonomization  in  this  section,  we  are  also  going  to  propose  an 
aid  to  the  process,  when  attempting  to  accomplish  certain  ends, 
in  the  next  section.  For  the  reader  interested  in  further 
discussions  than  is  provided  in  these  two  sections,  reference  is 
made  to  Finley  and  Muckier  (ref.  6). 

TAXONOMIZATION:  WHAT  IS  IT?  Taxonomization  is  the  process  of 

first  collecting  things  of  interest  together  and  then  finding 
some  identification  and  organization  of  these  things  which  will 
lead  to  further  understanding  and/or  will  make  these  things 
manageable  in  some  way.  It  is  usually  the  case  that,  with  the 
understanding  gained  from  the  first  effort  at  taxonomizing  at  a 
relatively  gross  level,  nore  detailed  and  complexly  structured 
taxonomies  are  subsequently  developed.  While  the  levels  most 
often  form  a hierarchical  structure,  this  is  not  necessarily  the 
case  (e.g.,  string  taxonomies  and  taxonomies  of  overlapping 
classes  located  by  ordinates  in  a multidimensional  space) . 


♦Taxonomy,  when  capitalized,  will  refer  to  the  science.  When 
uncapital.ized,  it  will  refer  to  a classification  system. 
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Examples  of  the  results  of  taxonomization  efforts  include 
the  kingdom,  phyla,  genera,  and  species  taxonomies  for  living 
organisms.  On  the  one  hand,  these  represent  attempts  to 
establish  a "natural"  or  evolutionary  order  for  the  organisms; 
on  the  other  hand,  the  resulting  taxonomies  also  assist  the 
organization  and  focusing  of  studies  on  organism  behaviors. 

Another  example  is  the  indexing  of  books  in  a library;  t1  is 
enables  the  librarian  to  both  evaluate  the  inventory  overall 
and  retrieve  individual  books  as  needed.  Yet  another  example  is 
the  development  and  application  of  a task  taxonomy  to  a collec- 
tion of  job  behaviors.  Here  the  purpose  might  be  to  develop  an 
information  base  that  can  be  studied  and  manipulated  as  needed 
to  design  a system  training  program  and  the  training  equipment. 

In  all  of  the  foregoing  examples,  it  can  be  noted  that  the 
development  and  application  of  the  taxonomy  was  actually  a way 
of  giving  additional  and  useful  meaning  to  collections  of  things. 
For  the  scientist,  it  is  usually  a matter  of  working  with  a 
particular  and  preselected  set  of  things  and  attempting  to  find 
that  classification  system  which  provides  some  "natural"  order 
based  on  properties  which  are  either  evolutionary  in  nature  or 
might  reflect  seme  scientific  principle.  For  the  practitioner, 
it  is  often  an  even  more  basic  process  in  that  the  first 
questions  to  be  addressed  often  concern  the  matter  of,  "Just 
which  sets  of  things  are  even  the  right  ones  to  look  at?"  (For 
example,  given  a system  development  program  and  limited  resources, 
which  of  the  operators,  operations,  and  equipments  should  be 
studied  in  detail  so  as  to  optimize  which  of  the  system  develop- 
ment and  operations  criteria?) 

Another  distinction  with  regard  to  the  practitioner  is  that 
he  is  taxonomizing  things  so  as  to  make  evident  those  properties 
relevant  to  solving  an  applied  problem  - rather  than  seeking  any 
"natural"  order  of  things.  Example  problems  might  be  ones  of 
designing  tasks  so  as  to  optimize  either  system  control,  system 
safety,  or  worker  satisfaction.  It  can  be  seen  that  each  of 
these  problems  concern  rather  different,  even  though  overlapping, 
subsets  of  all  the  properties  that  could  be  associated  with  a 
collection  of  tasks.  The  practitioner  must  develop  and  apply 
that  taxonomy  which  will  emphasize  those  task  properties  relevant 
to  the  problem  and  organize  these  properties  and/or  the  tasks 
themselves  in  a way  which  leads  to  problem  solution.  And  one 
thing  that  must  be  remembered,  but  too  often  is  not  - that  task 
taxonomy  which  yields  information  useful  for  addressing  one  of 
the  foregoing  task  design  problems  is  not  likely  to  yield  much, 
if  any,  information  for  solving  any  of  the  other  problems. 

In  summary,  taxonomization  is  the  process  of  developing  a 
taxonomy,  or  classification  system,  which  will  group  and  order 
things  so  as  to  give  them  greater  meaning  and  to  make  them  more 
manageable.  For  the  scientist  the  purpose  is  to  gain  knowledge 
about  the  things  studied  per  se,  while  for  the  practitioner  the 
purpose  is  to  gain  knowledge  relevant  to  the  solution  of  an 
applied  problem. 
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WHERE  IS  TAXONOMI ZATION  USED?  Taxonomization  is  used  throughout 
the  analytic  process,  wherever  it  is  necessary  to  organize  and 
identify  things  in  order  to  proceed.  Some  of  these  points  of 
usage  are  discussed  briefly  here  so  as  to  provide  examples: 

(1)  Populations  identification,  (2)  Systems  description,  (3)  Mea- 
surement scales  and  data  sampling,  and  (4)  Models  formulation. 

POPULATIONS  IDENTIFICATION'  OR,  DEVELOPMENT  OF  A POPULATIONS 
TAXONOMY  Given  a question,  the  first  matter  that  requires 
resolution  is,  "Just  which  populations  of  objects  and  conditions 
do  we  need  to  be  concerned  with?"  As  indicated  in  Figure  2, 

Tables  1 and  2,  and  on  page  6,  this  is  a prerequisite  to 
identifying  what  models  and  samples  of  the  real  world  are  to  be 
studied  and  what  measures  are  to  be  taken.  The  performance  of 
this  first  task  requires  that  the  analyst  or  practitioner  review 
the  operational  world  and,  in  effect,  group  and  organize  the 
components  thereof  in  terms  of  the  question.  The  result  of  this 
effort  is  usually  both  a gradual  restatement  of  the  question  and 
an  organization,  or  taxonomization,  of  the  world  until  the  one 
can  be  mapped  into  the  other.  The  effort  will  be  successful, 
i.e.,  the  question  will  be  answerable,  to  the  extent  that 
populations  of  objects  (e.g.,  systems,  operators,  behaviors)  and 
conditions  (e.g.,  scenarios)  can  be  identified  which  are  directly 
relevant  to  the  question  and,  also,  are  valid  and  meaningful 
samples  of  the  real  world  for  extrapolation  purposes. 

SYSTEMS  DESCRIPTION  The  foregoing,  populations  identification, 
is  the  first  and  an  iteratively  performed  step  of  systems 
description,  where  the  purpose  is  to  formally  define  a set  of 
taxonomies  and  apply  them  to  the  populations  that  have  been 
selected  for  study.  The  act  of  describing  the  system  and  system 
environments  of  concern  is  one  of  applying  the  descriptive  and 
analytic  caxcnomies  that  have  first  been  formulated  "in  the  head". 
The  process  of  preparing  for  these  description  activities  is  one 
of  taxonomization;  the  results  of  the  preparation  process  are 
the  system  and  task  taxonomies  used  for  system  description 
purposes . 

MEASUREMENT  SCALES  AND  DATA  SAMPLING  The  process  of  defining 
the  measures  to  be  used  in  a study  involves  several  steps  which 
are  either  based  on  the  results  of  taxonomization  or  else  require 
taxonomization  to  accomplish  them.  The  initial  efforts  at 
defining  measures  involves  an  examination  of  the  populations 
determined  to  be  relevant  to  the  problem  (or  available  for  study 
at  least)  and  a determination  of  what  measures,  if  any*,  ought  to 


*No  measures  may  be  taken  if,  for  example,  it  is  decided  to  use 
sampling  procedures  such  that  the  population  can  be  assumed  to 
be  "representative"  and  no  information  about  the  effects  of 
population  differences  is  desired. 
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be  taken  on  each  of  these  populations.  As  the  determination  of 
what  constitutes  the  study  relevant  populations  is  a taxonomiza- 
tion  procedure,  the  initial  efforts  at  defining  measures  is 
based  on  the  results  of  taxonomization. 

Two  later  steps  are  the  determinations  of  (1)  How  should 
the  measurement  scales  be  bounded  and  divided?*,  and  (2)  How 
should  the  resulting  scale  segments  be  sampled?  The  first 
determination  is  a matter  of  scale  definition  in  terms  of  study 
purpose  vs.  measurement  capabilities  and,  like  populations 
definition,  is  a taxonomization  question.  The  second  determina- 
tion results  in  a data  sample  taxonomy  of  sorts  where  the 
categories  are  defined  in  terms  of  quantity  relationships  (i.e., 
the  sample  N taken  at  each  of  the  "factor  levels")  and  randon  vs. 
fixed  sampling  definitions,  all  of  which  serve  to  determine 
statistical  procedures  and  the  conclusions  that  can  be  drawn 
from  analysis  results  (cf , ref.  23)  . 

MODELS  FORMULATION  As  noted  in  Figures  2 and  3,  and  Table  1, 
system  model  contents  and  operations  formulation  is  the  final 
"putting  together"  of  all  the  pieces,  measures  and  descriptions, 
resulting  from  the  previous  steps  in  the  analysis  process , As 
such,  the  goodness  with  which  it  can  be  accomplished  is  very 
dependent  on  the  completeness,  validity,  and  relevancy  of 
preceding  steps.  And  the  goodness  of  accomplishment  in  turn 
fully  determines  the  extent  to  which  useful  information  can  be 
gained  in  subsequent  analyses  where  the  formulated  models  and 
data  inputs  are  exercised. 

HOW  DOES  ONE  DO  IT?  In  the  beginning  we  noted  that  there  were 
two  aspects  to  this  business  of  grouping  and  ordering  collections 
of  things:  (1)  the  rules  of  assignment  and  distinction  for 

taxonomies  and  (2)  the  process  of  taxonomization.  The  first 
aspect,  as  then  noted,  has  been  the  main  province  of  Taxonomy. 

A good  overview  of  what  Taxonomy  can  presently  provide  to  the 
analyst  is  given  by  Sokal  (ref.  19).  A summary  is  as  follows: 

a.  Mathematical  tools  r ’eriving  a posteriori  taxonomies. 
Examples  of  well-knov,.  techniques  include  factor 
analysis  and  cluster  analysis  methods. 

b.  In  effect,  a data  bank,  or  library,  is  available  for 
reference  purposes.  This  library  contains  al.L  the 
already  developed  taxonomies. 


*E . g . , should  phenomena  in  the  temperature  range  of  40°  to  100° 
or  60°  to  80°  be  investigated?  And  should  the  "factor  levels" 
studied  within  these  bounds  represent  divisions  of,  for  example, 
10°  (e.g.,  60°,  70°,  and  80°)  or  5°  or  1°? 
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c.  Principles  for  the  structuring  of  taxonomies.  These 
range  from  mutually  exclusive  classes  without  order  to 
hierarchically  ordered  mutually  exclusive  classes  to 
overlapping  classes  located  by  ordinates  in  a multi- 
dimensional space. 

d.  Principles  for  classification  procedure,  i.e.,  rules 
for  operating  a classification  system.  Examples  include 
monothetic  vs.  polythetic  classification. 

The  second  aspect,  taxonomization,  is  perhaps  best  escribed 
as  an  ability  which  can  be  improved  upon  through  recognition  of 
its  existence  and  evaluations  of  the  taxonomies  resulting  from 
its  operation.  We  don't  know  how  one  actually  goes  about  "doing 
it",  but  any  practitioner  knows  how  useful  the  right  taxonomy  for 
the  job  is;  and  how  worthless,  if  not  dangerous,  the  wrong 
taxonomy  is.  And  scientists  clearly  know  that  progress  in  an 
area  is  first  evident  in  and  is  dependent  on  the  development  of 
a usable  taxonomy. 


i 
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SYSTEMS  TAXONOMY  MODEL 

Practitioners  and  analysts  involved  in  working  on  systems 
problems  often  complain  that  few  research  findings  appear  to. 
have  any  relevance  or  utility  for  their  system  problems.  The 
fact  is,  at  least  with  manned  systems,  despite  the  amount  of  so 
called  systems  work  that  has  been  done,  very  little  exists  in 
the  way  of  system  level  understanding  and  knowledge  (cf,  refs.  12 
and  13).  Actually,  a principal  reason  for  this  condition  is  a 
rather  obvious  one  - few  studies  ever  include  in  their  measure 
sets  parameters  of  input,  state  characteristics,  or  output 
performance  and  effectiveness  at  both  the  component  and  that  system 
levels  of  description.  For  example,  au  noted  by  Meister  with 
regard  to  studies  of  "man-machine"  systems,  researchers  often 
study  parameters  related  to  the  man  (e.g.,  training,  attitudes, 
operator  performance) , occasionally  study  parameters  of  the 
machine  (e.g.,  display  size),  but  very  seldom  study  parameters  of 
the  system  (e.g.,  layout  and  coordination  of  the  system 
components,  system  level  performance)  (ref.  13) . And  a review  of 
systems  analysis  reports  quickly  leads  to  the  conclusion  that  the 
same  tendencies,  in  reverse  order,  are  true  for  these  kinds  of 
studies  (cf , ref.  21) . As  the  practitioner  and  the  analyst  need 
information  on  all  of  these  in  combination ; the  components,  the 
overall  system,  and  the  component-system  relationships,  studies 
which  omit  the  system  or  one  of  the  major  components  (e.g.,  the 
man  or  the  machine)  - i.e.,  investigate  only  part  of  the  problem  - 
do  not  provide  the  necessary  information. 

As  discussed  in  some  detail  in  Finley  and  Muckier  (ref.  6), 
a most  basic  reason  for  the  foregoing  problem  appears  to  be  the 
failure  on  the  part  of  researchers,  and  on  the  part  of  practi- 
tioners and  analysts  too,  to  realize  the  explicit  existence  of 
both  systems  and  their  components  as  separate  and  distinct 
entities?  entities  which  constitute  separate  populations  of  N>0, 
populations  that  can  be  sampled  and  measured  so  that  conclusions 
related  to  systems  and  to  system-component  relationships  can  be 
drawn . 

As  previously  noted  in  Tables  1 and  2,  one  of  the  first 
steps  in  the  design  of  a data  base  analysis  program  or  an 
empirical  experiment  is  to  identify  the  populations  of  objects 
and  conditions  to  be  worked  with.  And  as  noted  in  above  discus- 
sions this  identification  of  populations  is  a taxonomization 
kind  of  step,  resulting  in  a problem  oriented  populations 
taxonomy,  and  is  an  essential  prerequisite  to  developing  a 
comprehensive  set  of  measures  for  data  collection  and  analysis 
purposes.  As  an  example  of  what  is  meant  by  "identification  of 
populations",  consider  a hypothetical  system  reliability  problem: 
the  mean  time  between  failure  (MTBF)  rates  for  an  aircraft  weapon 
system  are  generally  higher  than  they  should  be  and  higher  at 
some  air  bases  than  at  others.  For  a problem  of  this  sort,  the 
populations  taxonomy  might  include  the  following: 
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Flightline  maintenance  systems 
Shop  maintenance  systems 
Maintenance  equipments 
Technician  crews 
Individual  technicians 
Technician  and  crew  tasks 
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Supply  systems  | 

Weapon  systems  j 
Weapon  system  subsystems  I 
Weapon  system  aircrews  1 
Individual  operators  j 
Operator  and  crew  tasks  j 
Weapon  system  hard  and  software  components  1 


Command  and  control  elements 

J 

Supervisors 

( 

Work  environments 

I 

i 

Forms  used  for  debriefing,  etc.  | 

n 

rj 

Missions  J 

Mission  environments  1 


On  each  of  the  above  populations,  the  investigator  would  have  to 
make  a decision  as  to  whether  to  measure  parameters  describing 
the  population,  or  to  control  these  parameters  in  samples  drawn 
from  the  population  to  a constant  value.  (e.g.,  work  only  with 
equally  and  highly-skilled  technicians).,  or  to  sample  from  the 
population  in  such  a way  that  the  sample  can  be  assumed  to  be  a 
representative  one  across  the  parameters  of  concern.  One  thing 
should  be  clearly  noted  in  the  above  taxonomy  of  populations: 
systems,  subsystems,  components,  behaviors,  and  system  environ- 
ments are  all  included.  And  it  is  only  by  such  an  explicit 
cognisance  of  systems,  etc.,  each  as  a separate  population,  that 
measures  will  be  taken  on  each  of  them,  sampling  procedures 
individually  considered,  and  then  relationships  drawn  between  the 
measures  - by  either  the  researcher,  the  systems  practitioner, 
or  the  analyst. 
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Having  said,  however,  that  the  populations  taxonomy 
developed  for  investigating  a systems  problem  needs  to  be  system 
oriented  and  to  include  the  system  as  a population,  as  well  as 
the  system  components,  does  not  make  it  an  easily  accomplished 
matter.  One  can  only  assume  that  if  it  were  easy,  it  would  be 
done  much  more  frequently  than  it  is.  The  above  list  is  the 
result  of  considerable  experience  with  that  kind  of  problem  plus 
a foundational  concept  to  be  offered  in  this  section:  the 

Systems  Taxonomy  Model. 

The  purpose  of  this  model  is  to  provide  a basis  and  tool  for 
developing  conceptualizations  of.: 

a.  Systems  as  entities  which  form  populations, 

b.  Populations  taxonomies  which  include  the  populations  of 
both  systems  and  system  components,  and 

c.  System  taxonomies  which  are  organizations  of  populations 
class  and  differential  characteristics  meaningful  for 
the  purposes  of  research  design  and  planning. 

The  discussions  presented  in  this  report  regarding  the  model, 
in  the  following  paragraphs,  provide  a brief  introduction  to 
model  background  concepts  and  to  the  model  itself.  For  a detailed 
discussion  of  the  viewpoints  and  concepts  which  form  the  founda- 
tions for  the  model  (populations  definition  concepts,  human 
factors  research.  Taxonomy,  and  situational  taxonomies),  of  the 
aodel  itself,  and  of  how  to  actually  use  the  model  for  forming  a 
systems  taxonomy  (i.e.,  dimensionalizing  the  system  entity  for 
purposes  of  identifying  system  and  component  populations,  and 
p ^ulation  characteristics, , the  reader  is  referred  to  Ref.  6. 

DEL  BACKGROUND  CONCEPTS  The  Systems  Taxonomy  Model  was 
developed  around  three  concepts:  (1)  Measurement  level  defini- 

tions, (2)  Levels  of  system  description,  and  (3)  Types  of 
question. 

MEASUREMENT  LEVEL  DEFINITIONS  When  taxonomies  are  considered  in 
the  abstract,  all  they  are  essentially  is  a set  of  measures  and 
measure  relationships.  And,  an  interesting  fact  about  measures  - 
and,  therefore,  about  taxonomies  - is  that  there  are  the 
measurement  levels  of  nominal,  ordinal,  interval,  and  ratio;  and 
that  these  levels  can  be  grouped  into  two  categories:  nominal 

which  includes  only  the  nominal  level  of  measurement,  and 
relative,  which  includes  the  ordinal,  interval,  and  ratio  levels. 

Nominal  measurement  systems  and  nominal  taxonomies  are 
essentially  the  same  thing:  a set  of  categories,  into  which 

objects  can  be  placed,  but  which  bear  no  necessary  relationship 
to  each  other.  An  Apples  and  Oranges  taxonomy  is  a good  example 
in  that  things  are' either  apples  or  they  are  oranges  and  no 
underlying  relationship  or  common  dimension  is  assumed. 
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Relative  measurement  and  classification  systems,  on  the 
other  hand,  consist  of  a different  kind  of  category,  or  measure. 
These  categories  are  dimensions  which  are  used  to  give  objects  a 
relative  value;  these  values  are  then  the  basis  for  ordering 
the  objects  with  respect  to  each  other.  Take,  for  example,  the 
interval  taxonomy,  or  set  of  measures,  provided  by  a five  point 
rating  scale  of  "goodness".  Objects  or  conditions,  once  assigned 
a number  from  this  scale,  can  be  grouped  into  one  of  five 
categories  and  will  then  have  an  order  relationship  to  all  other 
objects  to  which  a number  has  also  been  assigned.  A relative 
classification  system,  or  taxonomy,  consists  of  some  set  of  such 
taxa,  or  measurement  dimensions. 

The  thing  of  interest  here  is  that  the  nominal  systems  give 
us  a management  capability  over  things  in  terms  of  their  unique 
aspects,  while  relative  systems  give  us  a management  capability 
over  things  in  terms  of  their  relationships  to  each  other. 

Whether  one  capability  or  the  other  or  both  is  desirable  depends 
upon  one's  purpose.  Both  of  these  capabilities  can  be  used  to 
define  entity  characteristics  and  to  distinguish  between 
populations;  the  difference  is  what  kinds  of  characteristics  one 
wishes  to  deal  with  - nominal,  relative,  or  both. 

LEVELS  OF  SYSTEM  DESCRIPTION  Systems  can  be  described  in  a 
number  of  ways  but  one  which  is  both  reasonably  common  and  very 
suitable  here  is  the  one  of:  (1)  System  objectives,  (2)  System 

functional  purposes,  and  (3)  The  various  system  activities, 
characteristics,  and  requirements.  Of  interest  here  is  that  a 
listing  of  systems  by  objectives  tends  to  form  a very  large 
(perhaps  infinite)  nominal  classification  system  - e.g., 
navigation,  transportation,  health  care,  etc.  - where  unique 
information  is  given  about  each  system  but  not  much  is  said 
about  how  the  systems  are  similar  to  or  dissimilar  from  each 
other.  On  the  other  hand,  a listing  of  systems  by  characteris- 
tics tends  to  form  a relative  classification  system  - e.g.,  size, 
level  of  automation,  environmental  conditions,  etc.  - where 
considerable  is  said  about  how  the  systems  compare  to  each  other, 
but  their  uniqueness  is  not  made  obvious. 

TYPES  OF  QUESTION  Questions  are  not  only  asked  about  different 
copies  and  with  different  objectives  in  mind  (e.g.,  to  predict 
vs.  to  diagnose),  but  also  for  different  purposes.  Two  basic 
purposes  of  interest  here  are  those  of  fundamental  research  vs. 
those  of  applied  research.  The  answers  sought  by  fundamental 
researchers  are  the  more  general  ones,  ones  applicable  to  systems 
in  general  with  some  knowledge  of  the  impact  of  major  system 
differences.  The  answers  sought  by  the  applied  researcher  are 
ones  specific  to  a particular  system  and  problem  at  hand.  From 
the  standpoints  of  the  practitioner  and  the  analyst  who  wishes  to 
extrapolate,  the  most  useful  documentation  is  that  which  is  an 
optimum  mix  of  both  the  general  and  the  specific,  both  the 
fundamental  and  the  applied.  Which  kind  of  answers  one  achieves 
is  most  basically  determined  by  the  kinds  of  taxonomy  one  starts 
out  with;  i.e.,  what  one  identifies  as  the  populations  and  popu- 
lation characteristics  about  which  the  study  is  to  be  concerned. 
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THE  SYSTEMS  TAXONOMY  MODEL  Given  the  possibility  of  an  infinite 
number  of  possible  system  problems,  it  is  entirely  reasonable  to 
conceive  that  an  infinite  number  of  different  populations 
taxonomies  and  populations  characteristics  taxonomies  also  exist 
for  the  solution  of  these  problems.  This  is  simply  because  the 
most  useful  taxonomy  of  any  sort  is  one  that  is  very  specifically 
tailored  to  the  information  needs  of  the  particular  problem.  Be 
that  as  it  may,  however,  it  is  still  possible  that  a general 
form,  or  model,  exists  within  which  all  of  these  taxonomies  would 
fit.  If  the  general  model  could  be  known  then  it  seems  reasonable 
to  expect  that  this  knowledge  would  facilitate  the  development  of 
problem-specific  populations  taxonomies  - and,  consequently,  any 
other  activities  which  are  closely  dependent  on  taxonomy  develop- 
ment, such  as  specification  of  the  relationships  between  measures, 
i.e.,  the  MOE  hierarchy. 

In  Figure  5 the  beginnings  of  such  a model  are  presented. 

As  listed  in  the  second  column  of  Figure  5,  the  Systems  Taxonomy 
Model  consists  of  three  major  levels,  distinguished  as  follows: 

a.  System  objectives  - the  reasons  for  a particular  systems 
existence; 
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b.  System  functional  purposes  - that  which  it  must  achieve 

to  some  level  of  adequacy;  and  ,5 

.1 

i 

c.  System  characteristics:  Structural,  Operator/Equipment,  \ 

Operating,  and  Support  Requirements  - how  the  system  is  | 

to  or  does  operate.  j 

, J 

The  definitions  of  these  three  model  levels  include  a relation-  j 

ship  to  the  nominal  vs.  relative  levels  of  measurement.  This  '' 

relationship  is  given  through  the  association  of  column  two  with  J 

column  one  in  Figure  5.  Examples  of  the  kinds  of  taxonomic  i 

categories  or  dimensions  that  might  be  associated  with  each  of  h 

the  model  levels  are  given  in  column  three.  i 


The  model  is  to  be  used  to  form  systems  populations,  systems  j 
characteristics  taxonomies,  and  systems  subpopulations.  Detailed  : 
directions  on  its  use  are  given  in  Ref.  6.  Suffice  it  to  say  j 
here  that  the  user  would  select  the  highest  level  (Level  One  is  , 
the  highest  level,  Level  Three  the  lowest)  needed  to  obtain  j 
information  on  his  particular  problem  and  that  the  resulting  \ 
taxonomies  would  then  be  based  on  that  top  level  plus  each  of  the  J 
lower  levels.  The  extent  to  which  the  lower  levels  are  used  will  j 
depend  on  whether  the  question  under  investigation  is  simply  a | 
status  question  or  is  instead  a predictive  or  diagnostic  question.  j 
As  an  example,  suppose  that  the  analyst  wished  to  gain  predictive  I 
and  diagnostic  information  regarding  the  achievement  of  specific  J 
system  objectives;  in  this  case,  the  analyst  would  wish  to  start  | 
at  Level  One  of  the  model  and  include  all  of  the  lower  levels.  | 
In  the  interest  of  performing  studies  which  will  gradually  form  a 1 
systems  and  system-component  relationships  information  base  use-  j 
ful  co  analysts  and  practitioners  in  solving  applied  systems  i 
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problems,  it  is  recommended  that  the  researcher  always  start  at 
Levels  One  or  Two  and  be  sure  to  include  all  "of  the  lower  levels 
(cf , Ref . 6) . 


SYSTEMS  DESCRIPTION:  APPROACHES,  COSTS,  AND  PAYOFFS 


The  purpose  of  system  description,  or  identification,  is 
spelled  out  in  Figure  2 and  Table  1.  It  is  to  provide  the 
systematic  knowledge  and  understanding  of  system  constitution  and 
operation  needed  for  effective  action  in  subsequent  analysis 
stages.  The  importance  and  use  of  information  provided  by 
systems  description  is  d'scussed  in  detail  throughout  this  report 
(cf,  e.g.,  pp.  21,  55-57,  67-68).  To  be  provided  here  is  a 
general  discussion  of  the  approaches,  costs,  and  payoffs 
associated  with  the  system  description  effort. 

APPROACHES  Three  issues  will  be  considered  here:  defining  the 

question,  collecting  the  data,  and  performing  the  analyses. 

DEFINING  THE  QUESTION  Given  the  question,  the  first  step,  as 
discussed  earlier  (p.  21)  is  to  identify  the  populations  that 
will  need  to  be  dealt  with.  An  aid  to  this  step  is  provided  in 
Table  1 where  the  items  requiring  description  are  broken  out 
according  to  the  possible  components  of  a question.  The  next 
step  is  to  develop  a set  of  descriptive  and  analytic  system  and 
task  taxonomies  which,  when  applied,  will  bring  out  the  informa- 
tion needed  to  address  the  question  and  organize  it  in  a useful 
manner.  The  application  of  these  taxonomies  results  in  the  data 
base  of  descriptive  materials  needed  for  measures  definition  and 
for  system  model  and  operations  formulation. 

COLLECTING  THE  DATA  As  discussed  in  Finley,  et  al  (Ref.  7) 
there  are  three  essential  sources  of  information  regarding  an 
operational  system:  observation  of  the  system,  the  system 

operators*,  and  system  documentation.  If  these  were  to  be  rank 
ordered  according  to  the  utility  and  amount  of  information  to  be 
gained  from  them,  the  system  operator  would  be  judged  to  be  one 
of  the  best  sources  while  system  documentation  would  be  judged  to 
be  the  least  useful.  Which  is  not  to  say  that  one  would  wish  to 
depend  on  only  a single  source.  If  at  all  possible,  all  three 
sources  should  be  used  in  an  integrated  manner.  This  will  afford 
the  maximum  data  from  each  (one  can,  for  example,  gain  much  more 
from  system  observation  if  one  already  has  the  working  knowledge 
that  can  be  gained  from  documentation)  and  a basis  for  judging 
the  validity  and  completeness  of  data  gained  from  each  source 
(each  operator,  for  example,  has  some  perceptions  of  his  system 
unique  unto  himself) . 

It  was  noted  in  the  first  report  (Ref.  7)  that  the  common 
practice  of  performing  system  description  analyses  based  on  just 
system  documentation,  without  inputs  from  both  system  observation 


*As  will  be  spelled  out  in  the  next  section,  concerning  operator 
models,  the  term  operator  should  be  understood  here  to  include 
members  of  both  the  "plant"  and  the  C&C  element. 
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and  operators,  often  produced  incomplete  and  erroneous  descrip- 
tion materials.  Based  on  our  own  program  work  with  a Navy 
system,  the  Carrier  Air  Traffic  Control  Centers  (CATCCS) , and 
subsequent  efforts  to  produce  descriptions  of  it,  that  original 
contention  has  been  substantiated  and  underscored:  the  produc- 

tion of  valid  and  useful  information  regarding  complex  and 
dynamic  manned  systems  requires  observation  of  the  system  and 
interaction  with  its  operators  - as  well  as  the  use  of  system 
documentation . 


PERFORMING  THE  ANALYSES  As  will  be  discussed  further  in  the 
paragraphs  below,  descriptive  analyses,  the  application  of 
system/ tas;  description  and  analytic  methods,  is  a costly  and 
time-consuming  process.  One  must  do  it  if  one  is  to  make 
informed  decisions  in  subsequent  analysis  stages  - but  one  must 
proceed  carefully  or  the  whol^  budget  for  analysis  will  be  shot 
and  the  desired  information  will  still  not  have  been  acquired. 


As  will  be  noted  in  a later  section  (pp.  50-51) , there  are 
several  general  methods  and  an  infinite  variety  of  problem- 
tailored  modifications  of  these.  Based  on  what  one  wants  to 
find  out  (Table  1 again)  one  selects  a subset  of  these  methods, 
tailors  them  to  the  question  and  the  system  under  investigation, 
and  applies  them  sequentially  to  the  system  until  the  necessary 
and  sufficient  information  base  has  been  developed.  The  results 
stemming  from  application  of  the  first  method  provide  some  inputs 
for  application  of  the  second  method,  etc.  It  is  suggested  that 
the  most  cost-  and  information-effective  way  to  proceed  is 
carefully  and  iteratively.  As  one  gains  more  knowledge  about 
the  system  and,  as  a consequence,  about  the  question  being  asked 
of  the  analyst,  it  is  often  the  case  that  a reapplication  of  a 
method,  with  some  modifications  or  to  a different  part  of  the 
system,  will  provide  additional  and  better  information.  What  is 
being  suggested  here  is  that  once  the  initial  selection  of 
description  methods  has  been  made  and  the  detailed  taxonomies 
constituting  them  have  been  initially  developed,  that  relatively 
inexpensive  trial  applications  of  the  methods  be  made.  The 
results  of  these  trial  applications,  and  data  on  the  cost  of 
performing  them,  then  need  to  be  reviewed  by  the  description 
analysts  and  the  subsequent  users  of  the  materials  being 
produced  to  see  if  the  desired  information  is  being  created  and 
whether  the  costs  will  be  commensurate  with  the  budget.  Changes 
can  then  be  made  in  the  methods  and  taxonomies  being  applied  to 
the  system  so  that  cost  and  information  criteria  will  indeed  be 
met.  If  the  manned  system  is  complex,  the  question  an  important 
one,  and  the  analysis  budget  limited,  it  is  generally  best  to 
iterate  through  such  an  application- test  and  evaluate-modify- 
reapply  cycle  more  than  once. 

COSTS  AND  PAYOFFS  One  fact  is  that  descriptive  analyses  are 
expensive  and  time-consuming  to  perform.  Even  more  expensive, 
however,  and  perhaps  dangerous  as  well,  is  the  performance  of 
subsequent  analyses  without  a valid  and  sufficient  set  of 
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measures,  models,  and  scenarios.  The  result  is  no  information, 
useless  information,  or  worse,  wrong  information.  Given  this, 
the  analyst  must  make  a decision  as  to  how  much  of  his  budget 
should  be  dedicated  to  system  identification.  No  final  answers 
(e.g.,  30%  of  the  budget)  can  be  given,  but  considerations 
important  to  that  decision  can  be  identified:  (1)  The  costs  of 

performing  the  description  analyses  vs.  the  cost  of  inadequate  or 
wrong  answers  to  the  questions,  and  (2)  The  resources  available 
for  collecting  the  data  and  performing  the  analyses  vs.  the  value 
of  valid  and  sufficient  answers.  An  iterative  analysis  procedure, 
as  described  above,  will  permit  the  analyst  to  optimize  across 
these  considerations. 


HUMAN  OPERATOR  MODELS:  THEIR  USE  IN  C&C  ANALYSES 

There  has  been  a great  deal  of  exasperation  expressed  by 
analysts  who,  in  the  middle  of  a systems  analysis  or  development 
program,  are  searching  for  ready-made  human  operator  models  which 
they  can  just  "plug  in"  to  the  system  model  - or  which,  by  a 
quick  and  easy  exercise,  will  provide  them  with  the  answers  to 
their  immediate  problem.  The  complaint  is  that  none  of  the 
existing,  ready-made  models  seem  to  fit  the  problem.  That  is, 
they  don't  include  input  and  output  terms  which  are  relatable  to 
terms  included  in  the  system  model  and/or  they  don't  concern  the 
functions,  tasks,  or  aspects  of  performance  which  seem  to  be  the 
critical  ones.  And,  to  the  extent  that  the  analyst  has  carefully 
gone  through  a system  description  process,  so  that  he  truly 
understands  the  unique  characteristics  of  that  system  in  terms  of 
this  problem,  this  is  ever  more  likely  to  be  true.  This 
situation  is  not  really  very  surprising,  however,  to  anyone  who 
realizes  two  things:  (1)  that  while  general  answers  to  general 

questions  provide  very  helpful  guidance,  it  is  still  nonetheless 
true  that  specific  questions  require  specific  answers,  and  that 
these  answers  must  usually  be  obtained  by  means  specifically 
tailored  to  that  problem;  and  (2)  the  complexity  and  variety  of 
human  components  and  of  the  operations  they  can  and  do  perform 
in  systems.  It  should  also  go  without  saying,  however,  that  a 
knowledge  of  existing  models  is  very  helpful  and,  indeed, 
necessary  if  one  is  not  to  keep  rediscovering  the  wheel 
unnecessarily.  First  of  all,  there  is  always  the  possibility 
that  there  is  indeed  a ready-made  model  in  existence  which  can  be 
used  with  little  modification  or  further  development.  But  even 
if  an  altogether  new  model  must  be  developed,  a knowledge  and 
understanding  of  existing  models  is  an  invaluable  resource  of 
ideas  and  provides  the  basis  of  understanding  needed  to  start 
the  effort. 

Given  the  foregoing,  there  seemed  to  be  two  ways  in  which 
this  program  might  assist  the  analyst  in  dealing  with  human 
operator  models.  One  way,  of  course,  was  to  develop  ideas 
concerning  how  to  actually  use  these  models  in  C&C  analyses.  The 
other  way,  an  outgrowth  of  the  foregoing  statements,  was  to 
develop  aids  for  the  analyst  in  selecting  and/or  developing  a 
human  operator  model  for  his  program.  We  will  discuss  the 
latter  problem  first. 

THE  SELECTION  AND  DEVELOPMENT  OF  OPERATOR  MODELS  As  befits  the 
complexity  of  humans  and  the  variety  of  systems,  there  are  an 
enormous  number  and  variety  of  human  operator  models.  This 
collection  of  models  would  be  a tremendous  resource  of  knowledge 
and  ideas  to  the  analyst  if  they  were  organized  in  some  fashion 
so  as  to  be  reviewable  „n  terms  of  analyst  information  needs 
and  if  there  were  some  guidelines  available  on  how  to  narrow  and 
direct  one's  field  of  search.  Some  trial  efforts  were  made  in 
this  program  with  respect  to  both  of  these  needs. 
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With  respect  to  guidelines  for  narrowing  one's  field  of 
search,  efforts  were  first  made  to  organize  stages  of  analysis 
(see  Chapter  II)  and  to  relate  these  to  question  topic  components 
Next  the  realm  of  operator  models  and  model  terms  were  organized 
in  terms  of  question  subject:  matter.  This  latter  organization  of 
models  was  tnen  mapped  into  classical  subject  areas  found  to 
represent  the  literature.  One  of  the  classical  subject  areas  was 
then  selected  for  a trial  effort  in  developing  a models  resource 
document  which  the  analyst  could  use  for  review  and  reference 
purposes.  These  efforts  are  each  discussed  further  in  the 
following  paragraphs. 

ORGANIZATION  OF  MODELING  ACTIVITIES  WITH  RESPECT  TO  QUESTION 
TOPIC  COMPONENTS  As  discussed  in  Chapter  II,  the  analysis 
process  was  broken  into  several  stages  of  analysis.  Three  major 
stages  - systems  description,  measures  definition,  and  the 
formulation  of  system  model  content  and  operation  - were  then 
related  to  question  topic  components.  These  relationships  are 
presented  in  Table  1,  page  9. 

ORGANIZATION  OF  MODELS  AND  THEIk  TERMS  WITH  RESPECT  TO  QUESTION 
SUBJECT  MATTER  A desirable  breakout  of  operator  models  is  one 
which  reflects  the  different  classes  of  question  subject  matters 
an  analyst  might  be  concerned  with  and  then  implies  different 
classes  of  dependent  and  independent  variables  for  dealing  with 
each  of  these  question  classes.  Such  a breakout  was  arrived  at 
during  the  first  year  of  program  effort  (Ref.  7,  p.  3°)  and,  at 
this  point,  it  still  seems  to  be  the  most  useful  one  for  analyst 
purposes.  This  breakout  consists  of  three  classes:  (1)  Opera- 

ting/Mission models,  (2)  Extended  Mission  models,  and 
(3)  Maintenance/Support  models.  These  classes  reflect  questions 
about:  (1)  mission  operations  and  design  per  se  where  time, 

when  considered,  is  used  as  an  operations  time  line,  (2)  extended 
and  repeated  missions  where  time  can  also  become  an  affect  or 
stress  factor,  and  (3)  personnel  maintenance  and  support  systems, 
where  time,  when  considered,  can  take  on  a personal,  as  opposed 
to  a mission,  lifetime  definition.  The  relationships  between 
these  three  classes  and  types  of  dependent  and  independent 
variables  are  presented  in  Table  3. 

A MAPPING  OF  THE  LITERATURE  AND  THE  DEVELOPMENT  OF  RESOURCE 
DOCUMENTS  Along  with  the  development  of  ideas  contained  in 
Table  3,  the  literatures  dealing  with  human  operator  performance 
were  reviewed  to  determine  how  they  might  best  be  categorized  at 
a general  level  so  as  to  make  them  accessible  to  and  reviewable. 
by  the  analyst.  The  thought  here  was  to  develop  categories 
representing  the  overall  and  traditional  makeup  of  the  literature 
to  map  these  categories  into  the  three  classes  of  models,  and 
then  to  develop  resource  documents  for  each  of  the  categories 
which  could  be  used  by  analysts  as  references  for  review  purposes 
The  categories  identified  as  representing  the  general  literature 
were:  ergonomics,  engineering  psychology,  industrial  psychology, 

motivation  theory,  personality  theory,  and  clinical  psychology. 
These  categories  were  mapped  into  the  three  classes  of  models 
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TABLE  3.  FACTORS  AND  OUTPUTS  ASSOCIATED  WITH  SYSTEM/ 
OPERATOR  PERFORMANCE  MODEL  CLASSES 


SYSTEM/OPERATOR  PERFORMANCE  MODEL  TERMS 

I 

. 

OUTPUTS  TO  WHICH 
PERFORMANCE,  COST, 

% 

■A 

SYSTEM/OPERATOR 

FACTORS  TO  WHICH  PERFORMANCE, 

AND  EFFECTIVENESS 

PERFORMANCE  MODEL 

SITUATION,  AND  ATTRIBUTE 

MEASURES  CAN  BE 

■j 

CLASSES 

MEASURES  CAN  BE  ASSIGNED 

ASSIGNED 

V 

Inputs  to  System/Operator 

Perception 

Processing 

i 

i 

j 

■4 

5PERATION/MISSION 

Decision-making 

Decisions  Made 
and  Actions 

MODELS 

Skills 
Abilities  . 

Performed 

Demands  on  Sy stem/Opera tor 
Mission  Conditions  & Events 

\ 

Changes  over  time  with 
respect  to: 

i 

Procedures 

SXTENDED  MISSION 

Skills,  Knowledge  Levels 
Adherence  to  Procedures 

Continuity,  Level 

1 

MODELS 

Attention  to  Details 

Changes,  and 

'i 

\ 

Fatigue 

Motivation 

Hostility 

Morale 

Variations  over 
Time  in  the 
Decisions  Made  and 
Actions  Performed 

1 

4 

.5 

Career 

Life  Style 
Standard  of  living 

■ I 

MAINTENANCE/ 
SUPPORT  MODELS 

Age 

Family 

Benefits 

Schedules 

Pay 

Continuity  and 
Level  Changes 
over  Time 
in  the 

Decisions  Made  and 
Actions  Performed 

l 

1 

Training  programs 

Messing 

Berthing 

Recreational  facilities 

I 


and  ranked  according  to  judged  relative  contribution  as 
follows: 


OPERATING/MISSION 

MODELS 


Ergonomics 
Engineering  psych. 
Motivation  theory 
Industrial  psych. 
Personality  theory 


EXTENDED  MISSION 
MODELS 


Motivation  theory 
Industrial  psych. 
Personality  theory 
Clinical  psychology 
Ergonomics 


MAINTENANCE/ 
SUPPORT  MODELS 

Industrial  psych. 
Motivation  theory 
Personality  theory 


One  of  the  better  developed  and  more  contained  of  these 
areas,  ergonomics,  was  then  selected  for  the  development  of  a 
trial  resource  document  and  the  result  is  cited  as  Reference  16. 
The  materials  in  this  document  are  organized  in  a way  which,  it 
is  hoped,  will  do  three  things:  (1)  serve  to  introduce  the 

ergonomics  view  of  man  to  the  analyst  who  is  without  background 
in  the  behavioral  and  biological  sciences,  (2)  make  the  spectrum 
of  ergonomics  models  apparent  to  and  reviewable  by  the  sophisti- 
cated analyst,  and  (3)  by  the  very  expliciteness  and  detail  of 
the  materials  make  it  apparent  that  ergonomics  depicts  only  a 
few  limited  aspects  of  the  human  operator  and  that  even  these  can 
be  very  complex.  We  are  rather  pleased  with  the  extent  to  which 
materials  were  pulled  together  and  organized  from  this  subject 
area  so  as  to  meet  the  needs  of  the  analyst  who  is  attempting  to 
integrate  the  ergonomics  aspects  of  "plant"  operations  into  an 
overall  system  analysis  program.  We  suggest  that  similar  efforts 
made  in  the  other  subject  areas  might  also  produce  useful  results. 

THE  USE  OF  OPERATOR  MODELS  IN  C&C  ANALYSES  In  an  earlier 
section  of  this  chapter,  the  functions  and  purposes  of  C&C  were 
spelled  out.  A distinction  was  made  between  operators,  as  well 
as  commanding  officers,  operating  in  the  C&C  vs.  the  "plant" 
modes.  It  was  then  pointed  out  that  it  is  the  responsibility  of 
the  C&C  element  to  manage  and  modify  the  plant  and  its  environ- 
ment to  the  extent  possible  so  as  to  reach  system  objectives 
within  resource  constraints.  The  impact  of  this  on  evaluate  of 
the  C&C  element  is  that  evaluation  of  C&C  performance  is  in  terms 
of  the  plant  and  the  plant's  environment;  while  evaluation  of 
C&C  effectiveness  is  in  terms  of  system  achievement  and  resource 
utilization.  Now,  when  we  say  that  "evaluation  of  performance 
is  in  terms  of  the  plant" , what  we  are  really  saying  is  that 
many  of  the  variations  that  can  be  made  in  the  terms  and  opera- 
tion of  a human  operator  performance  model*  are,  in  effect, 


♦and  models  of  the  other  system  components  as  well.  The  focus  of 
this  discussion  is  on  operator  models,  but  variations,  for 
example,  in  hardware  reliability,  are  also  reflections  of  C&C 
performance  and  need  to  also  be  considered  in  any  comprehensive 
evaluation  of  C&C. 
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representations  of  C&C  performance,  strategy  and  tactics.  That 
is,  that  if  we  vary  models  in  particular  ways  we  are  representing 
alternative  C&C  approaches,  i.e.,  C&C  performance . If  we  then 
exercise  these  different  versions  of  an  operator  model,  the 
resulting  changes  in  system  achievement,  if  any  and  in  view  of 
the  resources  expended, are  then  measures  of  the  effectiveness  of 
the  C&C  element's  strategy,  tactics,  and  performance. 

HOW  TO  USE  OPERATOR  MODELS  IN  C&C  ANALYSES  The  use  of  operator 
models  for  C&C  analyses  must  be  based  on  a thorough  understanding 
of  (1)  C&C,  (2)  the  particular  system  being  worked  with,  (3)  the 
relationships  between  C&C  and  operator  processes,  and  (4)  the 
ways  in  which  operator  models  can  be  varied  to  reflect  C&C 
performance  and  effectiveness.  The  foundations  for  understanding 
items  (1)  and  (2)  are  covered  elsewhere  in  this  chapter,  while 
the  methods  for  understanding  these  items  are  presented  in 
Chapter  V.  Items  (3)  and  (4)  constitute  the  subjects  for  this 
section. 

C&C  AND  PLANT  OPERATOR  RELATIONSHIPS  To  assist  the  under- 
standing of  relationships  existing  between  the  C&C  element  and 
the  operator,  performing  as  a member  of  the  system  plant,  a 
C&C/Human  Operator  Relationships  model  is  presented  in  Figure  6. 
The  model  consists  of  a C&C  block,  a human  operator  block,  and 
two  kinds  of  relationships  between  them:  a C&C  flow  and  an 

information  and  data  flow.  There  are  two  things  to  be  noted 
about  this  model.  One  is  that  the  C&C  and  operator  blocks  should 
each  be  understood  to  represent  modes  of  operation  rather  than 
necessarily  representing  separate  or  individual  people.  Any  one 
person,  or  teams  of  people,  may  at  times  operate  in  either  or 
both  modes.  The  important  consideration  is  that  the  performance 
of  an  operator  in  the  plant  mode  is  intimately  affected  by  the 
performance  of  that  same  operator  and/or  others  in  the  C&C  mode. 
The  main  concern  here  is  an  understanding  of  those  things  which 
can  be  directly  vs.  indirectly  modified  by  the  C&C  element  and 
of  what  kinds  of  things  the  C&C  information  system  can  get  data 
on. 

Second,  the  human  operator  model  contained  within  the 
operator  block  is  a more  general  and  comprehensive  one  than  most 
and  should  encompass  most  existing  operator  models.  It  should 
be  noted  that  the  model  not  only  includes  motives  and  levels  of 
aspiration  blocks,  it  also  includes,  as  intermediaries,  the 
levels  of  performance  possible  vs.  desired.  And,  most  important, 
that  the  levels  of  performance  desired  are  defined  by  inputs 
from  the  C&C  block  and  weighted  by  such  things  as  system 
responsibility  and  importance.  Further,  it  should  be  noted  that 
while  the  model  includes  levels  of  aspiration,  it  also  includes 
such  things  as  actual  vs.  possible  environments,  and  abilities 
and  skills,  as  inputs  to  the  determination  of  that  level  - and 
that  the  level  of  performance  achieved  is  a subsequent  output, 
with  the  ultimate  consequence  of  everything  being  the  setting  of 
attitude  (greater  endeavor  vs.  hostility,  etc.)  which  then  acts 
in  a feedback  loop  of  the  model.  The  development  of  the  overall 
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relationships  model  and  the  operator  model  was  based  on  the  | 

definitions  of  systems,  C&C,  and  system  plants;  our  observations  J 

of  the  relationships  actually  in  effect  in  operational  systems  1 

(principally  the  Carrier  Air  Traffic  Control  Centers,  or  CATCCs) ; 1 

and  a general  model  of  the  operator  developed  by  Ullrich  and  3 

rooted  mainly  in  motivation  and  subjective  utility  theory 
(Ref.  20).  j 

VARYING  OPERATOR  MODELS  When  it  is  said  that  the  C&C  j 

element  is  responsible  for  enactment  of  system  roles  and  for  f 

achievement  of  system  goals,  it  follows  that  the  C&C  element  is  | 

responsible  for  managing  the  states  and  activities  of  the  plant  | 

and  its  environment  such  that  these  things  can  be  accomplished  - i 

both  in  the  short  and  the  long  run  and  within  resource  constraints.  | 

From  this  it  follows  that  anything  which  can  be  varied  in  a human  ^ 

operator  model  - and  could  be  expected  to  vary  as  a function  of 
C&C  action  alternatives  - can  then  be  used  by  the  analyst  to  | 

reflect  actual  or  possible  C&C  strategies  and  tactics.  Each  | 

model  presents  its  own  special  case  of  possibilities,  but,  so  as 
to  provide  a better  idea  of  what  these  might  be,  some  of  these  j 

will  be  listed  and  then  what  could  be  done  with  a selected  sample  1 

of  four  models  will  be  discussed.  The  list  is  as  follows:  1 


(1)  Performance  standards  1 

Criteria  for  performance  j 

Prioritization  or  "essentiality"  ratings  assigned  to  tasks  > 

(2)  The  range  values  of  performance  variable  distributions  j 

The  shapes  of  performance  variables  distributions  j 

Parameter  values  I 

Each  of  the  above  m the  first  group  can  be  modified  or  maintained  ' 

by  the  direct  order  of  the  C&C  element.  Each  in  the  second  group  i 

can  be  expected  to  change  or  remain  the  same  as  a consequence  of  ) 

certain  command  actions.  For  example,  the  command  element  could  j 

change  the  training  or  staffing  quality  and  quantity  requirements  . -j 
and  this  would,  in  turn,  tend  to  modify  the  skill,  fatigue,  and  i 

motivation  levels  of  personnel  and,  consequently,  the  ranges  and  j 

distribution  of  their  performance  levels.  j 


( ) 


A MONTE  CARLO  MODEL  OF  TRACKING  BEHAVIOR  A remarkable 
study  was  reported  on  in  1963  by  Adams  and  Webber  (Ref.  1).  The 
focus  of  the  authors  in  the  report  was  on  the  validity  and  utility 
of  a Monte  Carlo  model  they  had  developed  of  operator  tracking 
behavior.  From  our  standpoint,  however,  the  model  itself  is  of 
least  interest.  What  is  of  considerable  interest  is  the  manner 
in  which  they  went  about  constructing  and  evaluating  their  model 
because,  in  this,  they  provide  (1)  a basis  for  evaluating  C&C 
and  (2)  a test  of  one  way  of  using  * available"  data.  With  regard 
to  C&C  evaluation,  the  authors  constructed  model  of  tracking 
behavior  where  an  ultimate  effectiveness  measure  is  seen  to  be  a 
function  of  a set  of  performance  measures  derived  from  operator 
time  and  error  scores.  These,  in  turn,  are  a function  of 
conditions . For  conditions  the  authors  chose  N = number  of  trials 
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(which  could  be  labeled  an  operator  characteristic,  in  this  case  | 
task  skill  level),  R and  I = regular  vs .• irregular  signal  inputs  | 
to  the  operator  (a  system  condition  or  mission  situation  kind  of  § 
variable),  and  x and  x,y  = one  or  two  dimensional  tracking  (a  1 
system  design  or  status  kind  of  variable) . These  kinds  of  condi-  | 
tions  could  each  be  a C&C  responsibility  in  a particular  system.  * 
If  any  of  them  are  indeed  a C&C  responsibility  in  the  system  j 
under  evaluation  then  the  C&C  level  of  performance  is  a function  k 
of  the  valuation  of  the  condition  terms,  modified  by  the  relative  j 
extent  of  their  impact  on  system  measures  of  effectiveness  (MOEs)  | 
and  cost.  Of  further  interest  is  that  the  authors  constructed  a j 
model  of  behavior  which  could  vary  as  a function  of  any  other  | 
kind  of  condition  which  could  also  be  reasonably  expected  to  j 
modify  tracking  behavior.  The  foregoing  conditions  were  simply  ; 
the  ones  selected  for  the  first  evaluation  of  the  model.  j 

j 

A final  note  of  interest  with  respect  to  C&C  evaluation  is  | 
that  because  the  ultimate  model  term  is  a system  measure.  Time-  j 
on-Target  (TOT)  in  this  case,  variations  in  this  valu';  which  I 
result  from  changes  in  conditions  caused  by  C&C  management  action  I 
are,  in  fact,  measures  of  C&C  effectiveness . i 
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Adams  and  Webber  also  provided  a test  of  the  possibility  of 
using  available  data  (see  Chapter  II  for  a discussion  of  what 
constitutes  "available"  data)  by  running  a series  of  experiments 
to  collect  performance  and  TOT  data  under  variations  of  the 
, aforenamed  conditions.  These  data  were  then  used  to  establish 

V alternative  model  term  values  and  functions  for  each  of  the 

different  conditions,  to  exercise  the  model  so  as  to  gain  model- 
generated performance  data,  and  to  evaluate  the  formulation  of 
the  model  they  had  developed.  The  two  concepts  they  established 
by  their  data  analyses  were  the  possibilities  of  different 
performance  variables  distribution  shapes  as  a function  of  such 
factors  as  skill  level  (fatigue  and  hostility  would  seem  to  be 
other  possibilities)  and  of  different  distribution  range  values 
as  a function  of  different  condition  values.  We  suggest  that  the 
analyst  can  similarly  use,  to  some  greater  or  lesser  extent  de- 
pending on  data  quality,  appropriateness  of  the  measures,  etc.* 
data  available  to  hi.m  for  the  purpose  of  formulating  and  exercising 
hypothetical  models,  evaluating  their  formulations,  and,  as  a 
consequence,  evaluating  C&C. 

1,  2 OPERATOR  SIMULATION  MODEL  Siegel  and  Wolf  (Ref.  18) 
have  developed  what  is  essentially  a procedures  simulation  model  for 
1 or  2 operators.  Task  performance  is  described  by  time  and  error 
values  and  procedural  performance  is  determined  by  mission  timeline, 
task  sequence  requirements,  and  an  "essentiality"  factor.  Task 
time  and  error  distributions  are  an  input  to  the  model  and  can 
therefore  be  varied  for  C&C  evaluation  purposes  if  so  desired  by 
the  analyst.  Similarly,  task  procedure  sequences,  which  could 
result  from  either  C&C  action  or  inaction,  can  be  modified  and 
. - . the  results  evaluated.  Of  special  interest  is  the  authors' 

v..y  concept  which  specifies  whether  or  not  successful  performance  of 

a subtask  is  essential  to  successful  completion  of  the  task. 
Nonessential  subtanks  can  be  ignored  in  the  simulation  during 
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"highly  urgent"  conditions.  >This  concept  is  somewhat  similar 
to  that  of  "prioritization" , where  the  command  element  establishes 
priorities  for  the  tasks,  functions,  criteria,  etc.,  within  the 
system  which  can  then  be  used  in  making  day-by.-day  control 
decisions.  By  changing  the  essentiality  ratings  in  a model  like 
Siegel  and  Wolf^,  we  are  then  reflecting  alternative  command 
strategies  and  tactics.  And  thus  an  exercise  of  the  model  using 
different  essentiality  ratings  provides  a means  of  evaluating  the 
effects  of  different  C&C  performances. 


QUASI-LINEAR  DESCRIBING  FUNCTION  MODELS  These  models  \ 

(cf,  Refs.  14  and  17)  represent  the  human  operator  as  a continu- 
ous describing  function  plus  a remnant  in  control  systems 
operation.  The  operator  variables  used  to  determine  describing 
function  values  include  gain,  reaction  time,  signal  anticipation,  ) 

and  averaging  variables.  Each  of  these  variables  can  be  expected  l 

to  vary  as  a function  of  training,  fatigue,  equipment  status,  } 

operator  ability,  and  planning  - all  of  which  can  be  considered  ] 

to  reflect  C&C  performance.  Therefore,  to  the  extent  that  C&C  \ 

could  possibly  modify  the  distributions  of  any  of  these  variables,  j 

then  the  changes  in  gain,  reaction  time,  etc.,  can  be  taken  to  ] 

reflect  C&C  performance.  And  the  effects  of  these  changes  on  ! 

the  model  output  terms  then  can  be  taken  as  a measure  of  C&C  1 

effectiveness.  '] 

\ 

C&C  ANALYSIS  MODEL  The  C&C  Analysis  Model  was  first  Ji 

developed  and  tested  as  a GPSS  language  model  of  CATCC  which  j 

simulates  air  traffic,  command,  control,  and  information  flows.  j 

The  discussion  here  will  be  a general  one  and  the  reader  should 
refer  to  pp.  60-68  and  to  Ref.  15  for  more  details  regarding  i 

the  model . 4 


The  quality  and  manner  of  operation  of  the  foregoing  flows 
are  understood  to  represent  the  results  of  alternative  system 
arrangements,  procedures,  operator  and  equipment  quality, 
command  mission  decisions,  situational  events,  etc.  The 
simulation  of  the  flows,  so  as  to  accomplish  the  recovery  of  a 
flight  of  aircraft,  determines  the  system  MOE  values  that  can 
result  from  each  of  the  alternative  flow  conditions.  The  C&C 
Analysis  Model  can  simulate  alternative  operational  scenarios, 
procedures,  and  conditions  to  that  level  of  detail  where  the 
simulation  can  still  be  expressed  in  terms  of  flow  effects. 

Some  flow  effects  can  be  understood  to  be  directly  the  result  of 
C&C  action  (e.g.,  the  aircraft  separation  criterion  selected  for 
that  recovery) , while  others  can  be  understood  as  being 
determined  by  C&C  (e.g.,  poor  control  due  to  training  or  fatigue 
problems;  or  missed  information  due  to  poor  radar  maintenance, 
operator  hostility,  team  management,  or  C&C  information  system 
design  problems) . These  effects  can  therefore  be  varied  to 
reflect  C&C  performance  and  the  results  analyzed  to  evaluate  C&C 
effectiveness . 

o 
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IV.  ANALYSIS  FOUNDATIONS:  DECISION 

CONCEPTS  AND  METHODS 


FOUR  HYPOTHESES 

In  the  analysis  and  evaluation  of  C&C  one  problem  is  not  a 
lack  of  raw  data.  Developments  over  the  past  decade  in  data 
acquisition  have  made  available  enormous  quantities  of  data. 

This  statement  is  not  meant  to  imply  that  all  data  acquisition 
problems  have  been  solved.  Nor  is  there  the  implication  intended 
that  all  relevant  data  are  readily  accessible.  But,  in  every 
case,  if  sheer  quantities  of  data  are  desired  they  can  be  found 
in  abundance. 

But,  more  data  are  not  necessarily  better.  Based  on 
experience,  four  hypotheses  of  C&C  data  analysis  may  be  proposed: 

H1:  Most  C&C  elements  demand  too  much  data. 

HjJ  Most  of  the  data  obtained  is  not  relevant  to  system 
control  and  planning. 

H^:  The  more  the  system  is  perceived  to  be  in  trouble,  the 

more  data  will  be  demanded. 

: The  more  data  demanded,  the  more  time  will  be  spent  in 

gathering  data  and  less  time  in  performing  functions. 

Some  comment  on  each  of  these  hypotheses  may  be  in  order. 

DATA  DEMANDS  (Hi):  Ready  availability  of  data  has  lead  to  what 

we  have  called  "an  orgy  of  data  acquisition"  (Ref.  7,  p.  18). 

There  is  a persistent  belief  that  quantity  of  data  will  somehow 
solve  problems.  Given  that  assumption,  it  follows  that  more 
data  will  solve  more  problems. 

In  fact,  one  could  argue,  based  on  recent  experience,  that 
more  data  may  lead  to  solving  less  problems.  There  is  a point  at 
which  the  analyst  and  decision  maker  can  receive  more  data  than 
it  is  possible  for  them  to  assimulate  and  process.  The  channel 
capacity  of  the  human  is  overwhelmed.  Beyond  that  point,  addi- 
tional data  lead  not  to  problem  solving  but  to  confusion.  A 
basic  limit,  therefore,  on  data  acquisition  is  the  amount  of  data 
an  analyst  can  receive  and  process  in  a reasonable  amount  of  time. 
It  is  to  be  suspected  that  the  human  limit  is  far  less  than  the1 
capability  of  present  data  acquisition  systems. 

RELEVANT  DATA  (H2)  '•  Most  available  data  are  irrelevant.  They 
serve  no  apparent  purpose.  Classic  in  this  regard  ‘s  the 
standard  application  blank  for  employment.  The  majority  of  the 
information  supplied  by  the  candidate  is  never  used  because  there 
is  no  conceivable  use  for  the  data.  The  data  are  not 
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relevant  to  the  decision  involved:  selection  and  placement  of 

the  prospective  employee.  Indeed,  data  relevant  to  that  decision 
are  often  not  present. 

For  C&C  element  control  and  planning,  most  decisions  depend 
upon  the  predi ction  of  events  to  come.  Control  and  planning 
require  anticipation  of  future  system  states  and  the  future 
actions  necessary  to  change  them.  In  short,  what  must  be 
estimated  are  things  to  come. 

Historical  data  are  useful  to  the  extent  that  the  system  is 
relatively  stable.  The  more  the  system  changes  the  less  valid 
predictions  based  on  these  data  will  be.  In  using  historical 
data,  the  analyst  must  also  estimate  stability  for  prediction 
purposes.  This  is  not  an  easy  analytic  task;  there  are  no 
formal  rules  to  assist  the  analyst. 

DATA  AND  SYSTEM  PROBLEMS  (H3) : When  a system  is  perceived  to  be 

in  trouble,  there  are  a number  of  possibilities.  The  system  may, 
indeed,  be  in  trouble  or  the  system  may  not  be  in  trouble. 
Problems,  if  they  exist,  may  not  be  those  of  the  analyst's 
perception.  It  is  interesting  how  often  system  "problems"  are 
perceived  based  on  vague  (and  sometimes  erroneous)  notions  of  the 
analyst  and  the  decision  maker. 

Whatever  the  case,  given  the  perception  of  a problem,  it  is 
almost  automatic  that  the  analyst  and  the  decision  maker  will 
demand  more  data  from  the  system.  Ostensibly,  this  is  a 
reasonable  step;  one  wants  more  data  upon  which  to  base 
corrective  action. 

But,  in  practice,  data  acquisition  may  confuse  rather  than 
clarify  system  problems.  The  problem  is:  data  about  what?  The 

analyst  and  the  decision  maker  should  interrogate  selectively. 

The  first  step  is  to  ascertain  what  the  problems  are  (e.g.,  the 
status  question;  see  pp.  48-50).  Only  then  should  detailed 
diagnostic  data  probing  take  place. 

A particular  difficulty  for  systems  in  trouble  is  "filtered" 
data.  Data  sources  may,  knowingly  or  unknowingly,  shape  data 
inputs.  These  "data"  may  either  hide  or  unnecessarily  accentuate 
system  problems.  This  tendency  places  a requirement  on  the 
analyst  to  obtain  validity  estimates  of. data  received. 

DATA  TIME  DEMANDS  (H4) : Very  few  analysts  and  decision  makers 

appear  to  realize  the  demands  that  data  acquisition  can  make  on  'a 
system.  Every  syst  n is  resource- limited,  and  it  is  imperative 
to  exercise  careful  control  over  resource  allocation  for  data 
acquisition. 

The  more  time  system  personnel  spend  in  data  generation  for 
evaluation  purposes  the  less  time  they  can  spend  in  performing 
system  functions.  C&C  (and  management)  elements  appear  to  be 


particularly  plagued  by  this  problem.  M*  .ny  middle-level 
managers,  for  example,  appear  to  spend  most  of  their  time 
reporting  rather  than  directing  and  performing  system  functions. 
If  this  is  a deliberate  allocation  decision,  then  it  is  accept- 
able. If  not,  one  must  accept  the  consequent  degradation  of 
system  performance. 

MEASUREMENT  UTILITY 


UTILITY  CONSTRAINTS  A fundamental  constraint  that  C&C  analysis 
and  decision  makers  must  accept  is  cost-effectiveness  in  data 
collection.  Every  data  point  must  be  evaluated  by  a number  of 
questions  before  the  data  are  collected:  What  question  will  the 

data  answer?.  Are  the  data  already  available  if  an  ADP  data 
store  exists?.  Can  the  data  be  collected?,  What  resources  must 
be  spent  in  collecting  the  data?.  If  acquired,  will  the  data  be 
worth  knowing?,  and  How  much  will  it  cost  to  collect  the  data?. 

Data  collection  is  not  a free  variable  in  the  system.  Data 
cannot  be  acquired  without  resource  expenditure.  Thus,  the 
analyst  and  the  decision  maker  must  be  bound  by  utility  con- 
straints in  data  processing.  Information  collection  is  essential 
to  the  survival  of  systems  and  the  operation  of  their  C&C 
elements.  But  excessive  data  processing  can  damage  the  system  if 
cost-effectiveness  constraints  are  not  placed  upon  system 
performance  measurement. 

MEASUREMENT  UTILITY  CONCEPT  Following  the  work  of  Cronbach  and 
Gleser  (Ref.  2) , the  notion  of  measurement  utility  can  be  made 
mathematically  explicit.  The  fallowing  equation  identifies  the 
parameters  th  t should  be  considered  with  respect  to  measurement: 
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utility  of  the  measurements  being  taken 

number  of  observations 

information  category 

probability 

treatment 

outcome 

value  of  outcome 

cost  of  collecting  the  measurements 


Of  basic  concern  here  are  the  payoff  functions  associated  with 
measurement  collection.  In  short,  in  any  given  case,  what 
measurements  are  worth  collecting? 


\ 


This  approach  stresses  a number  of  critical  aspects  to 
measurement  utility: 
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First,  what  measures  (y)  are  necessary  and  sufficient  to 
describe  the  process  being  measured?  A great  deal  of  experience 
has  shown  that  an  "obvious"  measure  often  does  not,  in  fact, 
provide  useful  information. 


Second,  what  sets  of  rules  are  assumed  as  constraints  on 
number  sets  (e.g.,  the  strategy  matrix  P+-/y  in  the  equation)? 

These  rules  are  often  used  to  sort  data  for  future,  more  detailed, 
evaluation.  They  are  particularly  important  if  adaptive  measure- 
ment systems  are  used. 

Third,  what  is  the  reliability  of  measurement?  Reliable 
(i.e.,  consistent)  measurement  does  not  guarantee  useful  informa- 
tion. However,  unreliable  data  will  lead  to  no  information 
whatsoever.  This  is  a necessary,  but  not  sufficient,  constraint. 

Fourth,  what  is  the  validity  of  measurement  (p^y  t)?  Do  the 
data  measure  what  they  say  they  measure?  A surprising  result  here 
is  that  useful  information  can  be  obtained  with  less  than 
completely  valid  data.  Indeed,  in  some  cases,  a degree  of 
validity  may  be  sacrificed  depending  on  desired  level  of  precision 
and  the  cost  of  the  required  information.  However,  some  minimum 
level  of  validity  must  be  obtained;  what  that  level  is  depends 
upon  the  specific  context. 


Fifth,  what  is  the  value  (ec)  of  collecting  the  information? 
Is  it  worth  knowing?  It  may  be  possible  that  the  majority  of  the 
measurements  yields  data  that  are  not  of  sufficient  value  to  be 
known . 


Sixth,  what  is  the  cost  of  collecting  the  data  (Cy)?  This 
cost  must  consider  not  only  the  direct  cost  of  the  resources  spent 
for  data  acquisition,  but  also  the  system  "cost"  in  resource 
allocation.  What  functions  are  not  being  done  when  data  collec- 
tion occurs? 


All  of  these  six  questions  must  be  asked  and  answered  if  the 
utility  - the  cost-effectiveness  - of  measurement  is  to  be  justi- 
fied. The  C&C  analyst  will  be  well-served  in  u^ng  these 
questions  to  guide  his  data  collection  process.  He  will  not  only 
obtain  better  data,  but  he  will  be  able  to  justify  the  utility  of 
the  data  he  has  collected. 
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V.  ANALYSIS  TOOLS:  CONCEPTS , METHODS,  AND  GUIDELINES 

FOR  THE  ANALYSIS  PROCESS 

WHAT  IS  THE  QUESTION? 

THE  QUESTIONS  OF  THE  ANALYST  Fundamental  to  the  analysis 
process  is  the  asking  of  relevant  questions  about  the  system  and 
the  components  of  the  system.  The  strategy  by  which  the  analyst 
asks  his  questions  will  have  enormous  impact  on  data  acquisition, 
processing,  and  interpretation.  What  the  analyst  is  trying  to 
do  is  to  turn  data  into  information,  not  random  information, 
but  information  directly  relevant  to  systems  and  subsystem 
questions . 

Unless  the  analyst  is  extremely  careful  about  this  process 
he  probably  will  incur  two  undesirable  results:  (1)  he  will  not 

get  meaningful  answers  to  the  questions  he  is  asking  and  (2)  he 
will  make  unreasonable  data  demands  on  the  system  and  system 
components.  Great  care,  therefore,  must  be  exercised  by  the 
analyst  in  his  search  for  information. 

THE  FUZZY  QUESTION  Most  frequently,  the  analyst  will  probably 
initiate  his  information  search  with  a fuzzy  question.  That  is, 
the  question  will  probably  be  vague  to  some  extent.  It  is 
assumed  that  he  may  not  know  exactly  what  he  is  trying  to  find 
out.  Even  when  the  analyst  thinks  he  has  a specific  and  well- 
defined  question,  the  question  may  not  be  the  one  he  should  be 
asking. 

Because  of  questions  that  are  fuzzy  or  irrelevant  to  the 
analyst's  search  for  information,  the  analyst  should  be  prepared 
for  a process  of  search  on  the  data  base.  At  least  two  steps  are 
involved:  (1)  he  must  be  refining  his  question  based  on  data 

obtained  and  (2)  he  must  be  checking  the  question  for  relevance. 
He  must  ask  himself:  "Is  my  question  clear?"  and  "Am  I getting 
an  answer  I can  use?"  At  best,  his  success  may  be  relative,  but 
that  is  usually  sufficient. 

A common  mistake  in  the  design  of  management  information 
systems  has  been  to  assume  that  all  possible  questions  can  be 
defined  a priori,  and  then  fixed  into  the  system.  If  this  were 
reasonable,  then  there  is  no  doubt  that  design  would  be 
simplified.  However,  in  practice,  what  has  resulted  is  a set  of 
simple  and  usually  irrelevant  questions  which  the  data  system  can 
answer  quickly  but  which  are  not  useful.  Unfortunately  (but 
understandably),  under  che  pressure  of  design  time  limits,  the 
question  set  is  usually  defined  too  quickly,  and  emphasis  is 
placed  on  questions  that  are  easily  structured  and  accommodated 
to  a data  base  design.  They  are  usually  not  the  questions  the 
analyst  needs  to  know  in  operations. 

The  information  system  designer,  therefore,  must  allow  for 
the  fuzzy  questions  which  the  analyst  will  be  asking.  The 
designer  should  consider  providing  search  aids  for  the  analyst. 

On  the  other  hand,  the  analyst  must  exercise  discipline  in  his 
search  or  the  data  demands  on  the  system  may  become  excessive 
(see  Chapter  IV) . 


THREE  FUNDAMENTAL  QUESTIONS  In  the  initial  technical  report 
(Ref.  7),  a taxonomy  of  three  basic  kinds  of  questions  was 
proposed:  (1)  system  status#  (2)  system  prediction,  and  (3) 

system  diagnosis. 


1.  The  system  status  questions  are  of  the  general  form: 

"How  goes  it?"  Most  usually,  they  are  measures  of  effectiveness 
(MOE)  about  the  total  system  and  not  about  system  components. 

The  existing  MOE  technology  appears  to  confuse  this  point,  and 
often  substitutes  component  MOEs  as  if  they  express  total  system 
performance.  They  do  not.  Optimal  subsystem  performance  often 
means  sub-optimal  total  system  performance.  Indeed,  for  optimal 
system  performance,  it  is  often  necessary  to  sub-optimize  the 
components.  For  example,  maximum  pay  benefits  for  personnel  may 
well  be  an  optimization  function  for  the  manpower  component  but 
it  is  hardly  optimal  (or  even  feasible)  for  the  total  system. 

2.  System  prediction  questions  search  for  the  future: 

"What  is  going  to  happen?"  Most  of  the  critical  questions  for 
the  C&C  component  will  be  of  this  class.  To  make  meaningful 
control  actions,  the  C&C  component  must  anticipate  system  events. 
Therefore,  it  must  make  estimates  of  what  might  happen  both  with 
and  without  alternative  management  actions. 

All  prediction  is  difficult  and  none  more  so  than  with 
manned  systems.  In  general,  the  farther  out  in  time  the  less 
valid  the  prediction  will  be.  The  analyst  surely  may  (and  often 
must)  ask  prediction  questions  in  the  five  and  ten  year  time 
spans.  But  he  must  be  extremely  careful  in  using  the  predictions. 
Further,  he  is  wise  not  to  ask  precise  and  detailed  prediction 
questions  about  time-distant  events. 

3.  System  diagnostic  questions  are  obviously  concerned 

with  the  question:  "What  is  going  wrong?"  To  ask  this  class  of 

questions  assumes  prior  status  and/or  prediction  answers. 

These  questions  begin  with  the  estimate  that  something  is,  or 
will  be,  inadequate.  The  analyst's  task  is,  then,  to  search  for 
the  causes  of  deficiency.  In  this  case,  the  search  probably  must 
extend  into  the  precise  details  of  component  performance. 


But,  the  analyst  must  be  alert  to  the  possibility  that  the 
system  is  not,  in  fact,  deficient.  Status  and  prediction  system 
estimates  are  often  global  sometimes  imprecise,  and  open  to 
misinterpretation.  The  lav.ter  is  particularly  true  when  desired 
system  performance  standards  are  not  made  explicit. 


For  example,  one  may  implicitly  assume  that  the  system  is 
100%  reliable.  For  human  systems,  this  appears  to  be  an 
unattainable  goal,  desirable  as  a goal  but  with  the  understanding 
that  something  less  will  be  achieved  in  practice.  What  the 
analyst  must  define  is  the  acceptable,  practical,  level  of  system 
reliability  that  is  adequate  for  system  performance.  Without 
that  value,  actual  reliability  data  obtained  from  status  ques- 
tions cannot  be  properly  evaluated. 
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Worse,  the  analyst  may  trigger  detailed  diagnostic  questions 
when  it  is  unnecessary.  The  analyst  must  realize  that  diagnostic 
questions  make  great  data  demands  on  the  system,  and  he  should 
only  ask  for  these  data  when  he  is  strongly  assured  that  he  needs 
the  data. 

FORMING  THE  QUESTIONS  Although  this  methodology  suggests  much 
flexibility  for  the  analyst  in  asking  system  and  subsystem 
questions,  it  is  essential  that  some  question  structure  be 
established.  The  analyst  must  be  given  some  assistance  in 
methods  of  question  search  and  of  tools  for  asking  those 
questions  which  can  be  structured  in  the  design  of  the  management 
information  system. 

There  are  many  ways  the  question  structure  and  process  can 
be  configured.  Experience  from  previous  systems  may  be  useful. 
Expert  opinion,  when  properly  collected,  will  be  fruitful.  More 
careful  analysis  of  the  potential  questions,  and  the  methods  by 
which  answers  can  be  obtained,  is  always  indicated.  For  the  C&C 
component,  analysis  of  potential  management  questions  is  rarely 
performed. 
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In  C&C,  for  example,  the  decision  maker  often  does  not  know 
exactly  what  he  should  ask.  When  he  wishes  to  know  the  status 
of.  the  system,  he  may  not  be  sure  how  to  ask  for  a status  answer 
in  terms  which  are  most  meaningful  and  timely  to  him.  If  some- 
thing is  wrong  (the  diagnostic  question) , he  may  not  know  the 
best  way  to  find  out  what  is  wrong.  Or,  in  either  case,  he  may 
ask  some  specific  set  of  questions  which  will  not,  in  the  end, 
give  him  meaningful  answers. 

Some  structure,  therefore,  is  essential  to  speed  and 
increase  the  efficiency  of  the  question  search  process.  How 
exact  that  structure  can  be  is  a function  of  many  variables. 

One  critical  variable  is  the  degree  to  which  the  system  process 
is  understood.  It  is  for  this  reason  that  so  much  emphasis  has 
been  given  in  this  report  to  taxonomizat ion  and  description.  We 
must  be  able  to  describe  the  system  if  we  are  to  understand  it. 
And  the  degree  to  which  we  are  able  to  describe  the  system  will 
determine  how  easily  (or  not)  we  are  able  to  structure  the 
questions  we  want  to  ask  about  the  system  (see  Table  1 for  a 
question-analysis  stage  relationship  structure) . 


SYSTEM  DESCRIPTION  METHODS 

Program  efforts  on  system  description  methods  were  for  the 
purposes  of  exploring  ways  to  describe  systems  for  C&C  analyses 
and  to  investigate  the  costs  and  payoffs  associated  with  these. 
Creative  efforts  resulted  in  an  integration  of  existing  methods 
into  a new  Operation/Mission  Requirements  Analysis  Method,  seen 
to  be  the  first  and  major  method  in  any  sequential  set  of 
methods  that  might  be  applied  to  a system.  This  method  was 
developed  and  evaluated  through  work  with  a Navy  system,  CATCC, 
and  is  reported  on  in  References  7 and  8. 
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Other  methods  were  also  tested  and  evaluated  in  this 
program  in  support  of  the  C&C  Analysis  Model  development  effort 
(pp.  60-68) . The  goal  was  to  determine  a set  of  methods  which 
would  satisfy  the  information  needs  for  system  model  and 
operations  formulation,  where  the  test  vehicle  was  the  develop- 
ment of  two  computer  models  of  CATCC.  Methods  found  to  satisfy 
CATCC  model  programming  needs,  up  to  a point,  were  the  afore- 
mentioned Operating/Mission  Requirements  Analysis,  Operational 
Sequence  Diagrams  (OSDs) , and  supplemental  descriptions  of  the 
information  contained  in  data  transmissions  and  of  decision 
processes  during  a sample  of  scenarios  containing  contingency 
events  which  modified  information  and  aircraft  flows  (e.g.,  radar 
breakdown,  bolter/waveof f aircraft) . An  additional  program 
output  resulting  from  these  efforts  was  a handbook.  The  handbook 
describes  the  above  methods  and  provides  details  on  the  work  that 
was  done  with  them  on  CATCC.  The  reader  is  referred  to  Gainer, 
Reference  8,  for  information  on  this  document. 

It  will  be  noted  that  the  phrase  "up  to  a point"  was  used 
above.  As  discussed  on  pp.  67-68,  the  computer  simulation 
language  used  by  the  C&C  Analysis  Model  is  capable  of  richly 
simulating  operator  behavior.  To  use  this  language  to  its  full 
capability  required,  however,  more  information  than  was  provided 
by  the  above  methods  or  could  be  provided  by  any  other  tradi- 
tional description  method.  As  noted  on  pp.  67-68,  it  would 
appear  very  worthwhile  to  develop  a new  description  method  which 
would  provide  the  programmer  with  the  additional  information 
needed  to  fully  use  computer  simulation  languages.  If  such  is 
not  developed  then  either  we  (1)  cannot  fully  analyze  the  effects 
of  the  plant  operator  or  of  the  C&C  element  on  overall  system 
effectiveness  and  performance,  or  (2)  the  programmer  makes 
modeling  decisions  based  on  inadequate  information.  Based  on  our 
efforts  with  CATCC  and  CATCC  program  development,  it  appears 
that  a description  method  could  be  built  around  the  General 
Purpose  System  Simulator  (GPSS)  language  and  that  such  a method 
plus  the  Operating/Mission  Requirements  Analysis  Method  would 
provide  sufficient  information  of  themselves;  that  isr  that 
the  additional  OSD  and  supplemental  description  method,  would 
not  be  needed. 


( ) 
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MEASURES  AND  MEASUREMENT 


As  stated  at  the  outset  in  Chapter  I:  "...understood 

meaning  is  the  goal  of  analysis  and... valid  and  sufficient 
measurement  - whether  qualitative  (e.g.,  verbal)  or  quantita- 
tive - is  the  means  to  that  end."  (page  3).  Each  piece  of 
data  collected  represents  a very  specific  piece  of  information; 
th j .rmation  content  is  known  only  to  the  extent  that  the 
me:,  v.re  represented  by  that  data  has  been  fully  and  validly 
defined. 


Given  the  foregoing,  it  seemed  to  be  most  important  that  a 
general  methodology  for  the  analyst,  like  thia  one,  be  oriented 
towards  the  problem  of  developing  a good  measures  set;  that  is, 
a set  that  will  provide  data  containing  the  information  needed 
to  answer  the  analyst's  question.  As  a consequence,  this 
methodology  has  stressed  the  taxonomization  and  systems  descrip- 
tion analysis  process  stages  because  they  lead  into  the  final 
definition  of  a measures  set.  In  this  section  we  will  introduce 
some  considerations  regarding  measures,  discuss  the  development 
process  a little  further,  and  discuss  the  final  selection  of 
measures  to  be  actually  used. 

MEASURES  CONSIDERATIONS  There  are  several  considerations  which 
bear  on  the  development  of  measures  definitions  but  which  do 
not  fall  easily  under  any  one  heading.  These  are  discussed  here. 

FUNCTIONALLY  DIFFERENT  TYPES  OF  MEASURES  Both  systems  analyst 
types  of  people  and  behavioral  psychologist  types  often  seem  to 
have  a rather  large  amount  of  difficulty  in  identifying  any 
measures  cf  the  human  component  which  bear  a relationship  to 
system  performance  or  effectiveness  measures,  and  vice  versa. 

I would  like  to  suggest  that  it  is  not  that  no  relationships 
exist  - they,  in  fact,  do  exist.  One  problem,  however,  is  that 
few  of  the  relationships  are  of  the  sort  being  sought,  while 
many  are  of  the  sorts  being  ignored;  and  that  those  ignored 
relationships  are  functionally  different,  usually  measurable, 
usually  analyzable,  and  important. 

Two  general  types  of  functional  relationships  between 
measures  are  of  interest  here:  (1)  The  categories  of  cause- 

effect  relationships  and  (2)  Time  relationships.  There  are 
three  categories  of  cause-effect  relationships: 

* Determining 

* Enabling 

* Bounding  (or,  Limiting) 


One  of  the  causes  of  the  problem  discussed  above  is  that  there  is 
a strong  tendency,  when  seeking  measures  of  the  relationships 
between  man  as  a "plant"  operator,  man  as  a member  of  the  C&C 
element,  and  the  system,  to  consider  only  those  relationships 
which  are  determining  in  nature;  that  is  measures  of  a man,  x, 
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which  determine  the  variance  of  a measure  y of  the  system.  Man 
also  acts,  however,  as  both  an  enabling  and  a limiting  agent  in 
most  systems,  performing  or  withholding  actions  so  that  other 
parts  of  the  system  are  enabled  to  operate  and  have  their  effect 
and,  similarly,  performing  in  a manner  which  will  either 
prevent  or  constrain  the  effects  of  actions  to  an  acceptable 
level.  When  the  operator  and  the  C&C  element  performs  in  these 
latter  capacities  a determining  equation  of  the  form  y = a + bx, 
where  x is  a measure  of  die  operator  and  y is  a measure  of  the 
system,  does  not  describe  the  action  of  x on  y.  Instead,  some 
other  form  of  expression,  as  in  calculus  or  a simulation  program, 
is  needed  in  order  to  describe  the  boundaries  wi*,hin  which  things 
are  constrained,  allowed,  or  enabled  to  operate  as  a function  of 
x.  Alx  of  which  is  simply  to  point  out  that  while  plant-C&C- 
system  relationships  of  a determining  nature  do  exist,  they  are 
not  the  only  ones  nor  the  only  important  ones.  The  plant 
operator  and,  especially,  the  C&C  element  also  perform  as 
enablers  and  limiters  - both  on  other  system  components  and  on 
the  system  as  a whole. 

The  second  type  of  relationship  of  interest  here  is  that  of 
time.  For  reasons  of  ease  and  simplicity  of  qualitative  and 
quantitative  expression,  no  doubt,  analysts  prefer  that  time  not 
be  a variable  in  the  equation  and  not  be  otherwise  considered 
except,  if  necessary,  in  the  form  of  a mission  timeline.  This  is 
reasonable  of  course  only  if  everything  that  the  operator  and  the 
C&C  element  does  has  an  immediate  effect  on  the  system  and  if 
his  effect  does  not  vary  as  a function  of  time.  This,  unfortu- 
nately for  the  analyst,  is  not  the  case  for  much  of  what  the 
operator  does  and  for  most  of  what  the  C&C  element  does.  (Take, 
for  example,  the  maxim  that  it  takes  two  years  to  feel  the 
effects  of  a new  manager;  or  the  contribution  of  the  maintenance 
technician's  adjustment  activities  to  system  reliability.)  The 
point  is  again,  as  in  the  foregoing  discussion  regarding  cause- 
effect  relationships,  that  there  are  many  important  time 
relationships  between  the  plant,  the  C&C  element,  and  the  system 
besides  the  immediate  online  relationship.  And  the  problem  of 
specifying  measures  of  the  operator,  C&C,  and  the  system  which 
will  relate  to  each  other  will  be  facilitated  if  the  analyst 
will  consider  those  relationships  which  either  include  time  as  a 
variable  or  else  themselves  vary  as  a function  of  time. 


COMPOSITE  VS.  MULTIPLE  MEASURES  It  is  often  the  case  that  data 
can  be  collected  on  a composite  measure,  Y,  and  on  some,  if  not 
all,  of  the  variables,  y,,  thought  to  be  a part  of  the  composite 
according  to  the  following: 

Y - a0  + aiYi  + a 2 y 2 + »•*  + amym* • 


( ) 


*The  additive  form  is  used  here  merely  as  a convenient  example 
and  its  use  is  not  intended  to  imply  that  it  is  necessarily  the 
proper  form  of  expression. 
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As  an  example,  Y could  be  a rating  job  "goodness",  while  the  y^ 
might  include  rate  of  pay,  a working  environment  rating,  a job 
interest  rating,  job  work  pace  parameters,  etc.  (cf,  e.g..  Ref.  11). 


Whether  or  not  one  wants  to  collect  data  on  just  Y or  also 
on  as  many  y^  as  possible  depends  on  several  considerations: 


a.  How  good  (i.e.,  valid,  sensitive,  and  reliable)  is  the 
measurement  da-f-a  on  Y?  And  how  completely  known  is  the 
real  definition  o c y? 


b.  What  is  the  question?  Is  the  question  strictly  and  for- 
evermore just  a status  question,  or  will  more  detailed 
prediction  and  diagnostic  questions  also  be  asked? 

c.  What  is  available  in  the  way  of  resources,  what  will  it 
cost  to  collect  data  on  additional  measures,  y^  and/or 
x^,  and  what  is  the  resulting  information  worth? 

HOW  WELL  KNOWN  IS  Y?  Very  often,  when  dealing  with  measures 
of  the  human  system  component,  it  is  very  difficult  to  be  sure  that 
the  measurement  data  are  truly  on  the  measure  we  originally  de- 
fined (validity,  sensitivity,  and  reliability  problems  arise  if 
they  are  not)  and/or  that  we  fully  understand  the  composition, 
i.e.,  the  detailed  definition,  of  Y.  In  the  event  that  there  is 
any  doubt  on  these  matters,  it  can  be  very  helpful  to  also  have 
data  on  measures  y.^  thought  to  form  even  part  of  the  composite  of 
Y.  If,  for  example,  data  could  be  collected  on  y2  of  the  above 
equation  at  little  additional  cost,  and  if  y2  was  thought  to  be  a 
substantial  part  of  the  definition  of  Y,  then  a regression  of  y2 
on  Y could  be  performed  and  evaluated.  If  the  regression  (or  cor- 
relation) proved  to  be  substantial  enough  then  one  would  have 
greater  confidence  in  both  the  data  on  Y and  the  definition  of  Y. 

If  the  regression  proved  to  be  minimal  or  in  the  wrong  direction, 
this  would  not  indicate  in  itself  wherein  lay  the  problem  - but  it 
would  raise  a flag  of  caution  in  either  using  Y data  for  other 
purposes  or  interpreting  what  the  data  on  Y actually  said. 


WHAT  IS  THE  QUESTION?  If  the  question  is  simply  one  regard- 
ing the  status  of  Y and  if  the  definition  of  Y is  clearly  known, 
then  all  that  is  needed  is  data  on  Y.  This  is  the  necessary  and 
sufficient  data.  If,  however,  the  question  is,  or  will  be,  a 
prediction  or  diagnostic  one  then  one  may  not  only  have  to  collect 
data  on  all  the  y^  possible,  but  also  on  any  determining  variables 
x^,  that  are  known: 


y - a„  + aiYi 


a2y; 


mm  o 


+ bjXx  + bkx2  + 


+ b x * 
n n 


The  reason  of  course  is  that  variations  in  the  y^  and  x^  terms  are 
the  causes  of  changes  in  Y,  comprising  the  question  in  prediction 


♦While  the  additive  form  of  expression  may  or  may  not  be  correct, 
depending  on  the  individual  case,  the  relationship  of  f (yx,  Y2, 
...,  y ) = f (x x , x2,  ...,  xn)  is  felt  to  be  the  correct  general 
case. 
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questions  of  "What  if  ...?"  and  comprising  the  answer  in 
diagnosis  questions  of  "What  is  the  problem?" 

WHAT  IS  THE  ANSWER  WORTH?  As  directly  discussed  in  Chapters 
IV  and  VI,  and  alluded  to  throughout  other  discussions,  one  does 
only  that  for  which  one  has  the  resources  and  which  is  worth  the 
cost.  Collecting  data  on  a measure,  be  it  from  an  operational 
or  test  environment  or  from  an  existing  data  base  of  some  sort, 
and  then  submitting  it  to  analysis,  is  an  expensive  process. 

On  the  other  hand,  coming  up  with  incorrect  findings,  because  one 
did  not  do  a sufficiently  thorough  job  of  measurement  and  analysis, 
could  also  be  very  costly.  If  the  cost  of  data  collection  and 
analysis  is  the  only  concern  and  resources  are  limited,  then  one 
should  deal  mainly  in  composite  measures,  that  is,  in  the  fewest 
measures  possible.  If  valid  and  complete  information  is  the  only 
concern,  then  one  should  deal  with  the  complete  set  of  both 
composite  and  multiple  measures.  Most  real-world  problems  require 
an  approach  that  is  a compromise  between  these  two  extremes  and 
the  trick  is  to  make  the  right  choices. 

EMPIRICAL  VS.  ANALYTICAL  APPROACHES  TO  MEASURE  DEFINITION  There 
are  two  quite  different  ways  to  approach  the  problem  of  developing 
a set  of  measures,  the  empirical  vs.  the  analytical.  Tha 
empirical  approach  is  the  classical  one  (in  the  job  evaluation 
and  test  development  literatures  at  least)  of  developing  large 
lists  of.  evaluation  items  and  then  subjecting  these  items  to 
standard  methods  of  validation.  The  approach  to  developing  the 
initial  list  is  essentially  a hit-or-miss  one  of  "if  it  moves, 
measure  it;  if  it  doesn't  move,  measure  it  anyway!"  The 
validation  process  is  an  empirical  one  and  very  expensive,  but 
if  the  right  measures  were  included  in  the  original  list,  they 
are  likely  to  be  identified  through  the  validation  process. 

The  problem  is  that  the  proper  measures  may  never  be  included  in 
the  original  list. 

The  analytical  approach  is  to  first  gain  an  indepth 
knowledge  of  the  system  through  systems  and  task  analyses,  and 
then  to  use  these  analysis  materials  as  one  basis  for  defining 
a set  of  measures.  The  cost  here  is  in  the  development  of  the 
initial  list  of  measures  - systems  and  task  analyses  are 
expensive.  The  constraint  is  that  the  development  and  applica- 
tion of  taxonomies  for  the  system  and  task  analyses  are,  in 
fact,  the  initial  settings  of  the  dimensions  around  which 
measures  will  later  be  defined.  If  the  taxonomies  are  not 
adequate  or  are  poorly  applied  then,  again,  the  proper  measures 
may  not  be  derived. 
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These  two  approaches,  empirical  and  analytical,  differ  in 
the  sources  from  which  the  measures  derive  their  validity  - 
empirical  test  vs.  an  analytic  knowledge  of  the  system.  Actually, 
to  the  extent  that  opportunity  and  resources  permit  and  that 
validity  is  essential,  both  approaches  should  be  taken.  That  is, 
the  development  of  the  original  set  of  measures  should  be  based 
on  a thorough  system  and  task  analysis,  while  additions  to  and 
validations  of  the  set  should  derive  from  empirical  test.  If 
obtaining  the  correct  and  complete  answer  is  worth  the  cost  then 
an  iterative  approach,  cycling  between  the  analytic  and  empirical 
stages,  is  best. 

THE  DEVELOPMENT  OF  MEASURES  As  shown  in  Figures  2 and  3 and 
listed  in  Tables  1 and  2,  the  analytic  approach  to  the  definition 
of  measures  is  an  evolutionary  process.  It  begins  with  the 
initial  taxonomization  of  subject  areas  (e.g.,  the  development 
of  a populations  taxonomy)  and  very  initial  identification  of 
what  general  kinds  of  measures  might  be  appropriate  (see  figure 
2) , proceeds  with  the  identification  of  the  system  through 
application  of  description  methods  (e.g.  , requirements  analysis 
formats,  system  and  task  taxonomies)  (see  table  1),  and  concludes 
with  the  final  evaluation  and  selection  of  those  measures  on 
which  data  can  be  obtained  and  which  will  provide  information  on 
entities,  operations,  and  relationships  such  that  the  analyst's 
question  can  be  answered  (see  table  1 and  figure  3) . 


Although  the  foregoing  may  seem  to  be  the  obvious  procedure 
to  some  readers,  these  readers  are  in  a minority.  Referring 
back  to  Figure  2,  the  more  usual  approach  is  to  rather  immediately 
jump  into  the  system  model  contents  and  operations  formulation 
.-tage,  using  whatever  measures  and  measurement  data  happen  to  be 
handy.  The  approach  outlined  in  Figures  2 and  3,  in  Tables  1 
and  2,  and  on  pages  6 through  12  is,  in  contrast,  a very 
conservative  approacn.  One  that  says,  if  the  system  is  a complex 
and  dynamic  manned  system  and  if  the  question  concerns  or  revolves 
around  the  C&C  element,  then  considerable  care  and  attention 
should  be  given  to  the  measures  set  development  stages  - the 
stages  prior  to  system  model  formulation  and  analysis.  The 
reasons  for  this  conservative  approach  are  simply  that,  under 
such  circumstances,  there  are  no  well-known  or  standard  measures, 
the  relationships  and  processes  that  should  be  measured  are  not 
the  easy  or  obvious  ones  (see,  for  example,  the  above  discussions 
regarding  cause-effect  and  time  relationships) , and  the  amount  of 
information  to  be  gained  from  subsequent  analyses  can  be  no  more 
than  that  provided  by  the  measures  set.  At  the  risk  of  being 
tiresome,  the  importance  of  the  measures  development  process, 
and  of  the  contributions  of  taxonomization  and  system  description 
to  it,  is  once  again  underlined.  It  is  a creative  process 
requiring  talent  to  perform  well?  but  it  can  always  be  improved 
upon  by  careful  attention  to  system  identification  and  the 
evolution  of  taxonomies.  The  adequacy  of  the  results,  that  is, 
the  amount  and  validity  of  information  provided  by  data  on  the 
measures  set,  determines  the  adequacy  of  the  answers  that  can  be 
determined  by  any  subsequent  modeling  or  analysis  efforts;  so 
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the  effort  to  develop  the  necessary  and  sufficient  measures  set 
is  worth  whatever  the  answer  to  the  question  being  asked  is 
worth. 


SELECTING  THE  MEASURES  TO  BE  USED  The  final  selection  of 
measures  to  be  actually  used  will  result  from  (1)  a comparison 
of  what  measures  are  wanted,  what  measures  are  represented  by 
the  available  data  (see  page  5 for  a definition  of  "available" 
data),  and  what  measures  can  be  taken  in  operational  or  test 
environments,  and  (2)  an  evaluation  of  the  costs  of  obtaining 
and  using  data  on  various  combinations  of  measures  vs.  the  value 
of  the  information  to  be  gained.  The  concepts  and  procedures 
for  item  (2),  the  utility  notions,  are  covered  in  Chapters  IV 
and  VI.  Here,  a brief  discussion  of  item  (1)  will  be  presented. 

A flow  between  the  question  and  the  set  of  measures 
selected  is  depicted  in  Figure  7 so  as  to  bring  out  the 
relationships  between  the  desired  measures  set  (i.e.,  the 
necessary  and  sufficient  set  of  measures  to  answer  the  question 
under  investigation) , the  already  available  measures  set,  and 
that  set  on  which  field  or  laboratory  data  can  be  collected 
(referred  to  here  as  the  "additional"  measures  set) . 


Figure  7.  Determination  of  the  Selected  Measures  Set. 
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What  is  being  said  here  is  simply  that,  based  on  all  the  inputs 
from  the  system  identification  exercises  and  from  such  founds- 
tional  materials  as  operator  models  and  the  C&C  Functional  Model 
(Figure  4),  the  analyst  will  develop  a list  of  desired  measures; 
that  is,  a list  of  those  measures  of  system  and  system  component 
states,  relationships,  and  operations  which  are  necessary  and 
sufficient  to  provide  that  data  which  can  then  be  used  to  answer 
the  analyst's  question.  Given  the  desired  set,  the  next 
questions  are,  What  is  already  available?  and  What  more  in  the 
way  of  additional  data  collection  is  reasonable? 

There  are  available  to  the  analyst  an  enormous  number  of 
data  bases  of  various  degrees  of  "formalization";  one  of  the 
most  informal,  and  often  most  informative  as  well,  is  the  system 
operator  himself.  The  more  formal  data  bases  are  those  contained 
in  computerized  data  banks.  The  problem  with  available  data,  be 
it  in  a hardcopy  file  or  a computer  file,  is  that  it  often  takes 
considerable  analysis  effort  to  determine  the  meaning  of  those 
data;  i.e.,  to  define  the  measures  which  those  data  represent. 

If  the  data  are  manned  systems  data  then  the  analyses  needed  to 
determine  the  available  measures  set  include  system/task  analyses 
regarding  the  system  in  which  the  data  were  collected;  these  may 
have  been  largely  completed  already  as  a function  of  determining 
the  desired  measures  set.  And,  of  course,  analyses  are  also 
needed  of  the  data  collection  instrument,  of  the  measurement 
procedures,  and  of  the  sampling  rates  and  time  with  respect  to 
other  events. 

Also  available  to  the  analyst  is  the  operational  environment, 
which  can  be  used  to  provide  data  on  an  "additional"  measures  set. 
This  availability  is  at  some  cost,  but  the  value  to  be  gained  is 
usually  well  worth  the  price  if  the  question  under  investigation 
is  important.  The  determination  of  what  measures  might  be  taken 
in  the  operational  environment  must  also  be  based  on  a system/ 
task  analysis  that  has  already  been  performed  and  was  itself 
based  on  operational  experience,  and  must  be  in  consideration  of 
the  measurement  capabilities  in  and  costs  to  the  operating  system. 
The  operating  system  will  present  some  set  of  measurement 
possibilities  unique  unto  itself,  it  cannot/will  not  brook  any 
interference  with  accomplishing  its  mission,  and  the  response  of 
its  personnel  to  any  form  of  questioning,  including  question- 
naires, will  be  a direct  function  of  how  sincere  the  analyst's 
quest  for  knowledge  appears  to  them  to  be;  i.e.,  is  the  analyst 
really  concerned  with  helping  resolve  their  problems  and/or  the 
Navy's  problems  and  does  he  appear  to  also  have  a reasonable 
insight  regarding  the  situation  such  that  he  might  be  expected 
to  have  some  degree  of  success? 

It  is  usually  the  case  that  the  desired,  available,  and 
additional  measures  sets  are  overlapping  sets;  the  analyst  can 
get  data  that  will  answer  his  question  of  course  only  to  the 
extent  that  the  desired  measures  set  is  overlapped  by  the  other 
two.  The  task  is  to  select  those  measures  from  the  "available" 
and  "additional"  measures  sets  which  are  either  also  members  of 


the  desired  measures  set  or  will  provide  some  estimate  of 
these  measures;  and  to  do  this  in  a manner  which  is  within 
costs  but  will  provide  as  much  of  the  necessary  and  sufficient 
data  as  possible.  The  making  of  this  judgment  will  be  assisted 
by  the  utility  notions  in  Chapters  IV  and  VI.  It  can  be  noted 
here  however  that  the  facts  that  the  desired  measures  set  may 
not  be  completely  covered  by  the  available  and  additional 
measures  sets,  and,  further,  that  costs  may  serve  to  constrain 
the  use  of  the  available  and  additional  measures  sets, 
establishes  a limit  on  the  extent  to  which  the  analyst's  question 
can  be  fully  answered  and  answered  with  known  validity. 
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A COMPUTER  MODEL  FOR  COMMAND  AND  CONTROL  ANALYSIS 

One  approach  to  the  evaluation  of  manned  systems  and  their 
C&C  elements  is  direct  empirical  observation;  however,  direct 
measurement,  and  especially  the  study  of  system  variables  by 
systematically  altering  conditions  within  an  operational  manned 
system,  are  often  impractical.  A model  of  the  system  which  allows 
variation  and  measurement  may  therefore  be  a cost-effective 
alternative,  and  consequently  a method  of  developing  system  models 
for  such  purposes  was  sought.  If  the  method  realized  at  this 
point  was  further  developed  and  refined,  so  that  it  could  be 
described  without  reliance  on  example,  it  would  constitute  a 
generic  computer  model  in  and  of  itself.  The  method  is  therefore 
herein  named  the  C&C  Analysis  Model. 

The  C&C  Analysis  Model  was  developed  through  the  selection 
of  what  seemed  to  be  the  most  suitable  kind  of  programming 
language  (a  simulation  language,  GPSS  in  this  case) , applying  it 
to  an  operational  system  (the  Carrier  Air  Traffic  Control  Centers, 
or  CATCCs) , and  developing  programs,  or  system  models,  of  this 
system  at  two  levels  of  complexity.  Since  the  purposes  of 
developing  the  two  programs  were  to  develop,  test,  and  evaluate 
an  analysis  method  and  to  develop  examples  for  method  exposition, 
the  programs  were  kept  as  simple  as  possible.  Sufficient  direct 
programming  experience  was  accumulated  so  as  to  provide  a basis 
for  the  establishment  cf  general  procedures  and  some  evidence  of 
the  workability  of  the  approach.  The  programming  exercises,  or 
CATCC  models,  also  provided  evidence  that  simulation  languages 
do  provide  the  means  for  common  computer  representations  of 
both  human  and  machine  components  so  that  subsystem  and  total 
system  performance  can  be  measured  and  integrated  in  terms  of 
common  computer  parameters. 

A complete  technical  report  was  produced  during  this  investi- 
gation of  computer  models  (Ref.  15)  and  in  it  were  discussed  the 
following  areas:  model  development,  guidelines  for  the  developer, 

and  discussion  of  the  various  uses  of  the  model  in  C&C  analysis. 

In  the  current  presentation,  however,  the  following  will  be 
emphasized:  (1)  the  characteristics  of  the  specific  GPSS  models 

of  CATCC  which  were  examined,  (2)  some  comments  about  the  task 
det  :ription  process  which  were  elicited  by  the  model  development 
process,  and,  (3)  the  relationship  of  these  models  to  the  overall 
C&C  analysis. 

CHARACTERISTICS  OF  THE  SYSTEM  MODELS  DEVELOPED  AND  EXAMINED  To 
be  described  in  the  following  paragraphs  are  the” characteristics 
of  the  selected  programming  language,  the  CATCC  system,  and  the 
two  models  of  CATCC  which  were  programmed  for  method  development 
purposes. 

THE  GENERAL  PURPOSE  SYSTEM  SIMULATOR  (GPSS)  LANGUAGE  The  compu- 
ter language  chosen  for  model  development  was  the  General  Purpose 
System  Simulator  language  (Refs.  9 and  10) . Based  on  a review 
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of  easily  available  languages,  the  simulation  languages  seemed 
to  hold  the  greatest  potential  for  C&C  analysis  purposes;  GPSS 
was  the  one  most  readily  available  to  the  investigators  for  test 
and  evaluation.  GPSS  is  a language  used  for  modeling  systems 
in  which  there  is  a flow  of  some  type  and  in  which  discrete 
events  characterize  the  state  of  the  system.  It  is  often  used 
for  simulation  of  such  things  as  traffic  flow,  assembly  line 
production,  and  the  formation  of  lines  (queues)  at  toll  booths 
and  ticket  offices.  It  seemed  especially  suitable  for  the 
simulation  of  the  information  flows  which  control,  result  from, 
and  are  affected  by  system  events  and  C&C  action. . 
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GPSS  is  a block-diagram  oriented  language.  When  a system  f 

b.1  :k  diagram  is  prepared  at  a sufficiently  molecular  level  using  | 

a ;PSS-specif ic  set  of  blocks,  the  computer  program  can  be  | 

derived  directly  from  the  block  diagram.  The  block  diagram  of  1 

a simple  queue  forming  at  a theatre  ticket  window  is  presented  ] 

in  Figure  8 as  an  example.  In  sequence,  the  block  diagram  i 

indicates  that  the  computer  model  should  (1)  GENERATE  trans-  ( 

actions  (people)  and  cause  them  to  be  introduced  at  intervals  \ 

according  to  a specified  distribution,  (2)  form  a QUEUE,  or  j 

waiting  line,  for  people  waiting  their  turn  and  keep  statistical  'j 

records  on  the  length  of  the  line  and  waiting  time,  (3)  SEIZE  a 
facility  (the  ticket  vendor)  when  an  individual  gets  to  the  front  \ 

of  the  line  and  the  ticket  vendor  is  not  busy,  (4)  DEPART  the  | 

queue,  (5)  ADVANCE  the  clock  according  to  a specified  distribution  j 

to  account  for  the  time  needed  for  the  ticket  to  be  given  and  \ 

money  exchanged,  . S)  RELEASE  the  facility  for  'e  next  person  in  * 

line,  (7)  TABULATE  statistics  (update  frequency  distributions)  of  | 

system  quantities  for  printout  at  the  end  of  the  computer  run,  and  j 

(8)  TERMINATE  the  transaction  (individual)  from  the  system.  This 
block  diagram  can  be  translated  into  a computer  program  along  with  J 

specific  system  quantities.  The  computer  model  can  then  be  | 

exercised  until  a specified  number  of  transactions  are  terminated;  j 

subsequently  the  run  would  stop  with  a printout  of  requested  j 

statistics.  \ 


GPSS  involves  a number  of  entities  which  are  included  in  a 
system  model  simply  by  referencing  them  by  number  (as  there  may 
be  many  of  each) . First,  transactions  are  entities  which  flow 
through  the  system  block  diagram.  Transactions  may  be  thought  of 
as  people,  automobiles,  airplanes,  mail,  etc.,  as  one  wishes. 

Each  transaction  carries  with  it  twelve  or  more  numbered 
parameters.  Values  associated  with  each  parameter  can  be  used  to 
characterize  the  transaction.  Facilities  are  entities  which 
simulate  the  processing  of  transactions,  with  one  transaction  at 
a time  being  processed.  Storages  may  process  (or  store)  a number 
of  transactions  at  a time,  but  a capacity  for  storage  must  be 
specified.  Queues,  as  already  indicated,  are  used  to  cause  the 
GPSS  system  to  maintain  statistics  on  lines  which  form.  Save- 
values  are  numbered  storage  areas  .-/here  special  data  may  be  kept 
until  the  end  of  a run.  Standard  Numerical  Attributes  (SNA)  are 
system  quantities  which  are  automatically  remembered.  These  and 
other  entities  are  available  to  the  GPSS  programmer  to  create  a 
computer  model. 
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THE  CARRIER  AIR  TRAFFIC  CONTROL  CENTER  (CATCC)  MODEL  The 
specific  system  selected  for  modeling  was  the  Carrier  Air  Traffic 
Control  Center  (CATCC)  (see  Ref.  7 for  a detailed  description  of 
CATCC) . As  may  be  seen  in  Figure  9 the  model,  has  five  kinds  of 
transactions  flowing:  (1)  the  aircraft  flowing  from  the  Marshal 

point  down  to  the  deck  of  the  carrier,  (2)  data  or  communications 
flowing  to  or  from  the  aircraft,  (3)  processing,  or  transforma- 
tion of  these  data,  and  (4)  requests  for  control  actions  flowing 
to  the  aircraft.  Additionally,  (5)  command  information  may  flow 
into  the  CATCC  from  external  sources. 

When  using  GPSS,  continuous  processes,  like  aircraft  flow, 
must  be  simulated  as  a discrete  approximation  of  the  continuous 
process.  This  requirement  for  discrete  approximations  was 
considered  to  be  acceptable  because,  however  continuous  a system 
process  or  the  information  regarding  it  may  be,  the  human 
generally  only  accepts  and  operates  on  discrete  samples  of 
incoming  information.  And  where  incoming  information  is  trans- 
mitted via  radar  or  verbal  communications,  as  in  CATCC,  the 
presentation  of  information  is  quite  discrete  in  nature  however 
continuous  the  underlying  process  may  be. 

During  each  discrete  path  segment  used  in  the  CATCC  model 
aircraft  errors,  fuel  depletion,  position  reports  and  control 
actions  were  updated  in  a cumulative  manner.  Two  GPSS  blocks 
are  instrumental  in  this  process:  the  SPLIT  block  transforms  a 

transformation  into  two  identical  transactions,  each  sent  along 
different  paths.  The  LOOP  block  returns  a transaction  to  the 
beginning  of  a series  of  blocks  until  the  transaction  has 
transversed  the  path  a specified  number  of  times.  In  this  way, 
a simulated  aircraft  is  caused  to  travel  the  specified  distance 
down  the  flight  path.  The  other  flows  are  simple  directed 
movements  without  looping. 

When  block  diagrams  are  generated  for  each  flow,  and  the 
programs  are  generated  and  executed  on  a digital  computer,  all 
types  of  GPSS  transactions  flow  "simultaneously"  simulating  an 
information  processing  management  system  in  which  transformations 
and  interactions  occur  in  the  same  event/time  relations  as  the 
CATCC.  The  GPSS  software  permits  record  keeping  and  the  calcu- 
lation of  measures  of  performance  and  effectiveness  as  the 
analyst  desires. 

THE  SIMPLE  GPSS  PROGRAM  The  initial  example  model  was  simple 
(for  general  block  diagram,  see  Figure  10) , having  been  developed 
to  allow  an  initial  test  of  GPSS  capabilities  on  the  CATCC  model 
structure  just  described.  As  simulated  aircraft  were  generated 
and  sent  to  the  Marshal  point,  each  was  assigned  a time  to  start 
the  approach  to  the  carrier.  Most  of  the  aircraft  were  spaced 
60-seconds  apart,  but  120-second  spacing  was  introduced  at 
regular  intervals  so  a3  to  create  "holes"  for  the  integration  of 
holter/waveoff  aircraft.  A single  "information  processor" 
tested  the  spacing  between  aircraft,  and  if  any  were  closer  than 
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Figure  10.  A Simple  GPSS  Simulation  (General  Block  Diagram) 


30  seconds  the  aircraft  would  be  sent  back  to  the  nearest  "hole" 
in  the  approach  pattern.  The  amount  of  time  and  fuel  were  proper- 
ly accounted  for  when  an  aircraft  was  diverted.  In  the  simple 
example  model,  the  approach  was  simulated  in  one  mile  flight  path 
segments . 

Each  transaction  (aircraft)  in  the  aircraft  flow  had  associ- 
ated with  it  a number  of  parameters:  flight  number,  flight  size, 

type  aircraft,  serial  number,  seconds  of  fuel  remaining,  clock 
time  storage,  airspeed,  heading,  glideslope,  checkpoint  (miles  to 
go),  holding  time,  and  clock  time  of  arrival  at  the  Marshal  point. 
The  value  of  each  parameter  characterizes  each  aircraft  and 
becomes  the  basis  for  identifying  and  controlling  information 
flowing  in  the  system. 

The  simple  model  was  tested  with  variations  in  the  number  of 
aircraft,  frequency  of  arrival  of  flights,  error  distributions, 
and  information  processor  rate.  For  each  computer  run  with  a 
given  selection  of  values  for  each  of  the  foregoing  model 
parameters,  a number  of  measures  of  performance  and  effectiveness 
were  automatically  computed  for  printout  at  the  end  of  each  run? 
these  included  frequency  distributions  for  airspeed,  heading, 
elevation,  information  processing  time,  aircraft  spacing,  aircraft 
transit  time,  and  recovery  time  (the  time  from  arrival  of  the 
first  aircraft  to  the  landing  of  the  last  aircraft) . These  tests 
indicated  that  the  C&C  analysis  model  structure  used  was  quite 
satisfactory. 

THE  EXTENDED  GPSS  PROGRAM  A second  expanded  example  was  develop- 
ed to  incorporate  additional  CATCC  features  to  a sufficient 
extent  that  development  of  the  full  model  of  the  system  could  be 
predicted  and  confidence  gained  in  the  analysis  method. 

Multiple  controllers  and  communication  channels  were 
included  in  the  expanded  model  (CATCC  includes  a Marshal  control- 
ler, two  control  teams  with  an  Approach  and  Final  controller,  and 
status  board  keepers) . Consequently,  handoff  procedures  were 
reflected  in  the  programming  to  divert  the  flow  of  information 
and  control  from  controller  to  controller.  All  tasks  and 
communications  were  appropriately  timed  in  the  model,  based  on 
assessments  made  from  operational  sequence  diagrams  and  task 
analyses. 

Incoming  information  about  the  position  of  aircraft  was 
stored  in  savevalue  locations  to  simulate  displays  which  could  be 
accessed  as  often  as  the  simulated  human  operators  needed.  The 
time  for  updating  status  boards  was  included  so  that  this 
information  was  appropriately  delayed.  Radar  displays  were 
simulated  in  a fashion  which  permitted  the  realistic  simulation 
of  dropouts  and  fadeouts  of  information.  As  the  type  of  control 
exerted  on  an  aircraft  will  depend  on  the  specific  circumstances, 
alternative  control  actions  were  included  in  the  extended  model, 
buc  neither  this  nor  any  of  the  previous  extensions  posed  any 
difficulty . 
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While  in  the  simple  example  each  operator  task  was  initiated 
by  some  external  transaction,  it  was  noted  that  many  CATCC  tasks 
are  operator  initiated,  or  are  continuously  performed  as  often 
as  time  permits.  At  a given  time  a number  of  tasks  may  be 
simultaneously  expected  of  an  operator.  The  operator  must 
therefore  timeshare  the  performance  of  these  tasks  in  some 
fashion.  A number  of  examples  of  timesharing  were  incorporated 
into  the  extended  example  model,  showing  the  capability  of  the 
method  to  handle  these  behaviors;  however,  it  was  difficult  to 
derive  the  required  task  description  to  support  such  programming. 
This  topic  will  be  subsequently  discussed  in  more,  detail. 

Human  operator  parameters  were  also  examined  in  the  extended 
model  to  determine  the  parameters  which  might  be  varied  to  reflect 
changes  in  human  performance;  these  included  processing  rates, 
types  and  rates  of  error,  and  the  manner  of  task  timesharing. 

In  brief,  the  extended  model  reflected  the  ability  of  the 
GPSS  language  computer  model,  the  C&C  Analysis  Model,  to  include 
the  salient  features  of  a man-machine  information  processing 
system  such  as  CATCC. 

HUMAN  OPERATOR  TASK  DESCRIPTION  The  manner  of  task  description 
within  the  GPSS-language  model  is  perhaps  one  of  the  model's 
greatest  strengths  and  might  well  be  developed  into  a new  task 
description  and  analysis  method.  The  GPSS  language  exhibited  a 
capability  for  rich  description  of  human  operator  tasks,  permit- 
ting a number  of  time- sharing  features  to  be  incorporated  with 
relative  ease.  For  example,  the  PREEMPT  block  permits  the 
imm.a-.ate  seizure  of  a facility  (the  human  operator)  by  a trans- 
action (information  to  be  processed) , with  the  built-in  feature 
of  returning  the  facility  to  whatever  it  was  doing,  picking  up  at 
the  point  of  interruption.  Further,  GPSS  permits  assigning 
levels  of  priority  to  transactions,  so  that  the  higher  priority 
transactions  are  serviced  first,  while  competing  transactions 


with  the  same  priority  are  serviced  on  a f irst-come-f irst-served 
basis.  Also,  with  some  additional  coding,  it  is  possible  to 
specify  that  a facility  process  transactions  in  specific  sequences, 
or  a little  of  each  transaction  according  to  a specified  order  of 
scanning.  These  features  were  all  tested  in  the  extended  model 
and  served  to  satisfy  the  needs  for  CATCC  simulation. 

As  discussed  in  pages  33-34  and  50-51  system  identification 
involves  the  application  of  a sequence  of  description  methods. 

One  employs  these  methods  iteratively  and  more  or  less  sequential- 
ly until  one  has  built  up  the  information  base  regarding  system 
and  operator  behavior  needed  to  specify  measures  and  formulate 
system  models  relevant  to  the  questior  being  asked.  The  develop- 
ment of  the  GPSS  models  of  CATCC  required  the  use  of  these 
description  methods  and,  as  expected,  they  supplied  the  informa- 
tion needed  by  the  programmer  in  the;  sequence  desired.  A problem 
ensued,  however,  in'  that  although  the  information  supplied  was 
necessary  information  and  in  the  desired  sequence,  there  was  not 
sufficient  information  regarding  task  behaviors  to  permit  full 
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use  of  GPSS  language  capabilities;  while  traditional  methods 
supply  much  of  the  information  regarding  system  and  task 
behaviors,  they  do  not  provide  information  on  the  timesharing 
and  prioritization  characteristics  described  above.  It  became 
clear  from  the  questions  asked  by  the  programmer  that  the 
sequence  of  description  methods  needed  to  be  expanded  and/or  the 
programmer  needed  to  closely  observe  the  operational  system 
himself.  It  appears  that  the  GPSS  language  could  be  the  means 
of  resolving  the  problem  of  incomplete  task  structure  description 
by  providing  a vehicle  for  the  expression  of  such  information. 

In  other  words,  a new  task  description  method  could  perhaps  be 
developed  around  the  GPSS  language,  the  application  of  which 
would  follow  the  application  of  other  methods  in  sequence  and 
use  the  information  base  supplied  by  them  as  inputs.  This 
possibility  is  further  discussed  on  pages  32  - 34  in  terms  of 
information  needs  vs.  cost  constraints. 

THE  USE  OF  MODELS  IN  COMMAND  AND  CONTROL  ANALYSIS  While  no 
extensive  tests  of  the  simulation  language  computer  model  were 
made  in  the  conduct  of  C&C  analysis,  examination  indicates  that 
these  models  should  be  generally  useful.  These  models  should  be 
useful  in  answering  systems  performance  and  effectiveness 
questions  since  measurement  of  system  performance  through  the 
model  is  unrestricted  and  such  performance  can  be  determined  as 
a function  of  many  variations  in  system  form  and  system  parameters. 
The  model  can  also  be  used  to  identify  potential  causes  of  system 
malfunction  and  means  for  correction,  again  through  noting 
performance  as  a function  of  variations  in  system  parameters  or 
structure.  New  measurements  can  be  attempted  and  tested  on  the 
model,  with  freedom  for  variation  in  the  form  of  the  measure 
until  the  desired  result  is  achieved.  The  model  structure  also 
provides  a testbed  for  the  development  of  human  operator  models 
in  a form  which  will  assure  that  human  performance  variations 
can  be  tested  along  with  corresponding  measurement  of  subsystem 
and  overall  system  performance. 

However,  such  models  are  not  the  beginning  or  the  end  of 
analysis.  Clearly  system  analysis  in  the  form  of  taxonomization, 
system  identification,  and  measures  specification  must  occur 
before  such  models  can  be  properly  constructed  (see  Figures  2 and  3) . 
When  analysis  skips  the  preceding  steps  and  begins  with  the 
generation  of  models  there  is  little  assurance  that  such  models 
have  any  relation  to  the  real  system  or  that  the  desired 
application  of  the  models  will  be  possible.  Model  development 
is  also  not  the  end  of  analysis,  for  empirical  testing  of  model 
results  is  necessary  to  establish  validity.  Computer  models  of 
the  type  described  here  are  useful  at  many  places  during  analysis, 
especially  as  a tool  for  answering  analytic  questions. 


INTEGRATIVE  DATA  USAGE 


Given  a set  of  measures  on  the  system,  the  question  arises 
of  how  to  integrate  them,  with  all  of  their  different  scale 
definitions,  etc.,  into  a data  analysis  package.  Some  of  the 
problems  and  considerations  involved  in  specifying  the  data 
analysis  program,  and  approaches  to  it,  are  discussed  here. 

MULTIPLE  MEASURE  SETS  In  assessing  data  available  from  systems, 
it  is  apparent  that  not  only  are  there  large  quantities  of  data 
• available,  but  the  data  can  be  acquired  from  many  different 
levels.  Figure  11  depicts  three  levels  of  measure  sets  that  can 
be  determined  from  the  personnel  component  of  a system:  system, 

task/job  and  behavior  measures. 

These  three  can  be  further  broken  down  into  six  kinds  of 
measures: 


a.  System  measures 

b.  Subsystem  measures,  including  those  of  the  C&C  element 

c.  Team  task  measures 

d.  Individual  task  measures 

e.  Attitudinal  measures,  and 

f.  Biophysiological  measures. 

Thus,  in  any  C&C  application,  we  may  be  confronted  with  at  least 
six  multiple  measure  sets. 

Some  comment  might  be  added  about  attitudinal  and  biophysio- 
logical measurement.  The  value  of  attitudinal  data  seems  to  lie 
in  the  fact  that  it  provides  certain  kinds  of  information  that 
no  other  type  of  measurement  can  provide.  Principally,  it 
provides  data  on  how  system  personnel  perceive  system  effective- 
ness. While  these  perceptions  may  not  be  correct,  if  they  are 
valid  they  are  an  attractive  cost-effective  source  of  data  on 
system  performance.  Even  if  the  perceptions  are  incorrect,  that 
fact  alone  makes  attitudinal  data  often  worth  collecting. 
Misperceptions  by  system  personnel  can  lead  directly  to  system 
performance  degradation. 

Biophysiological  measures  are  becoming  increasingly  valuable 
in  that  they  may  be  the  only  measures  which  provide  an  estimate 
of  the  "readiness"  of  the  personnel  element.  Some  give  indica- 
tions of  effort  expended  and  available  at  a given  time.  Past 
performance  measures  are  perhaps  not  so  good  an  indicator  of  these 
states . 

In  any  case,  the  multiple  measure  sets  and  the  data  sets  on 
them  provide  some  serious  questions  about  integrated  data  usage. 
Three,  in  particular,  are  difficult: 

First,  each  set  has  multiple  measurement  alternatives  within 
it.  For  system,  task,  or  behavior  measures  there  is  no 
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Figure  11.  Multiple  Measure  Sets  in  C&C  and  System  Evaluation. 
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standardization  of  measures.  Indeed,  among  the  many  measures 
available,  there  is  rarely  any  criteria  for  the  selection  of 
"best  measure" . The  literature  on  comparative  evaluation  of 
alternative  measures  is  a very  small  one.  Another  possible  source 
of  guidelines  could  be  quantitative  theory  where  measurement 
parameters  would  be  necessarily  specified.  No  such  theory  exists. 

Second,  the  various  measure  sets  probably  are  not  mathemati- 
cally commensurable.  They  are  usually  a combination  of  ordinal, 
interval  and  ratio  scales.  Recent  mathematical  investigations 
have  suggested  some  serious  potential  problems  in.  combining 
varied  types  of  scales  into  formal  equations.  These  problems  are 
particularly  pronounced  in  the  assignment  and  use  of  differential 
weighting  functions . 

Third,  Figure  11  implies  the  existence  of  formal  relation- 
ships between  system,  task,  and  performance  measures.  While  some 
results  have  been  produced  along  these  lines,  they  are  usually 
mathematically  weak,  correlative,  functions.  Although  informative 
and  useful  to  a degree,  they  must  be  handled  and  interpreted  with 
caution. 

THE  NECESSARY  AND  SUFFICIENT  SET  Figure  11  shows  that  the  end 
point  of  each  dimension  is  the  derivation  of  the  necessary  and 
sufficient  integrated  measure  set.  It  also  implies  a necessary 
empirical  step  in  selecting  integrated  measurement. 

Figure  12  presents  a diagram  of  the  empirical  process  of 
generating  integrated  multiple  measure  sets  for  C&C  evaluation. 

The  discouraging  implication  is  that  the  analyst  cannot,  without 
caution,  select  some  a priori  measure  set  and  expect  satisfactory 
results.  In  every  case,  the  analyst  would  be  wise  to  institute 
empirical  checks  on  the  integrated  measure  sets  which  he  intends 
to  use.  Fortunately,  this  can  often  be  done  while  the  analysis 
is  underway.  In  short,  however,  the  analyst  cannot  trust  fully 
a-.iy  known  integrated  measure  set. 

DATA  ALGORITHMS  Integrated  data  usage  implies  some  formal 
(mathematical)  or  informal  relationships  between  measures  and 
allowable  algorithms*  for  manipulating  data.  We  must  be  concerned, 
therefore,  about  acceptable  data  analysis  algorithms.  In  short, 
given  a set  of  measures,  how  are  the  data  points  to  be  manipulated? 
This  is  the  essential  question  of  integrative  data  usage. 

Some  of  the  many  problems  in  this  context  include  the  follow- 
ing: 
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♦which  may  not  be  necessarily  formally  mathematical.  Any  manipula- 
tion or  interpretation  of  multiple  data  sets,  however,  assumes 
some  algorithms,  explicit  or  unstated. 


(•' ) 


GENERATE  MEASURES 

TASK  ANALYSIS 
PERFORMANCE  PREDICTION 
QUANTITATIVE  MODELS 


DATA  COLLECTION 

ADVANCED  DESIGN 
TEST  & EVALUATION 
RESEARCH  & DEVELOPMENT 
PERSONNEL  SYSTEM 

DATA  SOURCES 

MANPOWER  POOL 

ORGANIZATION 

SYSTEM  MANAGEMENT 

SUPERVISION 

JOB/EQUIPMENT  DESIGN 

SELECTION  & CLASSIFICATION 

TRAINING 

ATTITUDE 

HUMAN-HUMAN  RELATIONS 
ENVIRONMENT 


THE 

CANDIDATE 

MEASURE 

SET 


MEASURE 

SELECTION 

TECHNIQUES 


THE 

NECESSARY 

AND 

SUFFICIENT 

SET 


MEASUREMENT 

DEVICES 


GENERAL 

PURPOSE 

SIMULATION 


f ) 


Figure  12 . 


The  Process  of  Generating  Multiple  Measure 
Sets  in  C&C  Evaluation. 


1.  The  terms  "data  algorithms"  and  "data  analysis  methods" 
are  often  taken  to  be  limited  to  traditional  descriptive  and 
inferential  statistics.  For  the  kinds  of  measurement  problems 
discussed  here,  these  tools  may  have  limited  application. 

However,  some  questions  could  be  put  in  hypothesis  testing  form 
appropriate  for  statistical  methods,  for  example:  Does  the 

system  status  at  this  moment  differ  from  a previously  predicted 
system  state?  The  answer  must  necessarily  be  "yes"  or  "no". 

Thus,  techniques  from  inferential  statistics  may  be  useful  for 
status  questions  provided  that  (1)  either  planned  or  desired 
states  can  be  quantitatively  expressed  and  (2)  a binary  answer  is 
sufficient  for  the  analyst. 

2.  The  analyst  may  ask  the  very  meaningful  question:  when 

have  sufficient  measurements  been  collected  to  provide  adequate 
information?  (Usually,  of  course,  the  answer  is  provided,  not 
by  measurement  consideration,  but  by  time  available  to  make  a 
decision.)  In  one  sense,  the  question  is  analogous  to  the 
sufficient  sample  size.  Assuming  we  know  the  right  dimensions  to 
measure,  how  often  must  we  measure  those  dimensions?  When  can  we 
stop  measuring? 


Although  there  are  techniques  available  to  handle  these 
problems  in  a formal  way  (e.g.,  Bayesian  models)  perhaps  one 
possible  option  should  not  be  ignored.  That  is  the  perception 
of  the  analyst  himself  that  he  has  sufficient,  credible,  data  to 
answer  his  question (s).  This  procedure  is  followed  more  than 
any  other;  it  would  seem  worthwhile  to  consider  training  analysts 
to  be  aware  of  systematically  executing  some  credibility  and/or 
"sufficiency"  criteria. 

3.  With  multiple  measure  sets,  it  is  impossible  to  avoid 
the  issue  of  the  differential  weightings  of  the  measures.  Many 
mathematical  techniques  are  available  for  optimum  assignments  of 
weights.  A pleasant  surprise  is  the  possibility  that  random 
assignments  of  weights  may  be  almost  as  good  for  decision  making 
as  optimization  (Ref.  3) . Some  limitations  can  be  expected. 

4.  Evaluation  is  expressed  in  some  set  of  criterion 
variables.  This  is  true  even  if  the  judgment  is  "good"  or  "bad". 
Unfortunately,  the  real  world  of  evaluation  is  never  that  simple; 
criterion  variables  abound  in  plenty.  One  example  is  the 
tremendous  numbers  of  measures  of  effectiveness  (MOE) . 

- From  the  standpoint  of  integrated  data  usage,  many  consider 
it  desirable  to  combine  MOEs  into  a single  measure  of  effective- 
ness. At  issue  here  is  the  longstanding  problem  of  multiple 
versus  composite  criteria.  A composite  criterion  is  simpler  to 
understand,  but  it  may  disguise  or  even  confuse  detailed  system 
performance  achievement.  The  answer  to  this  choice  rests  in  the 
question:  what  does  one  want  to  know?  This  question  will  be 

reconsidered  in  the  following  Chapter  (VI)  in  the  case  of  asses- 
sing the  operational  readiness  of  a C&C  element. 


mm 


5,  The  basic  system  question  being  asked  (e.g.,  system 
status,  prediction  or  diagnosis)  will  influence  presumably  the 
data  analysis  algorithms  that  will  be  required.  In  many  ways, 
it  would  appear  that  system  status  questions  are  perhaps  the 
easiest  (at  least  in  form)  to  answer.  The  complexity  of 
prediction  questions  will  vary  at  least  as  a function  of  the 
relative  stability  of  the  system.  In  short,  the  more  stable  a 
system  is  the  more  predictable  its  future  state  will  bo. 
Diagnostic  questions  on  the  other  hand,  may  pose  quite  different 
analytic  problems  and  call  for  rather  different  techniques. 

6.  Prom  the  basic  mathematical  and  operations  research 
literature,  we  have  available  a staggering  amount  of  analytic 
models  and  tools  from  which,  theoretically,  one  can  select.  One 
classification  distinguishes  deterministic  from  stochastic 
methods.  Deterministic  model  classes,  for  example,  include 

(1)  Linear  models,  (2)  Network  models,  and  (3)  Dynamic  models. 
Stochastic  methods  include,  of  course,  any  probabilistic  model 
form  which  assumes  distributions  rather  than  values  on  parameters. 
One  very  fundamental  question  is:  When  should  these  models  be 

used  in  C&C  data  analysis  methods?  Or,  given  a C&C  question, 
what  model  is  best  for  answering  that  question? 

The  answer  may  be  found  by  considering:  (1)  the  type  of 

question  the  analyst  is  asking,  (2)  the  precision  of  the  answer 
he  requires,  (3)  the  nature  of  the  data  available,  and  (4)  the 
stability  of  the  system  being  evaluated.  As  the  complexity  of 
the  question,  precision,  data  and  system  increase,  the  analyst 
must  proceed  from  linear,  deterministic,  models  to  non-linear, 
stochastic,  methods.  Utility  demands  that  he  select  the  simplest 
possible  model  even  at  the  sacrifice  of  some  depth  in  the 
obtained  answers.  The  formal  demands  - mathematical  and 
empirical  - of  complex  models  do  not  appear  to  provide  sufficient 
value  in  the  outcome  except  where  system  survival  is  at  stake. 
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COMMAND  AND  CONTROL  EVALUATION 

The  amount  that  one  needs  to  find  out  about  the  C&C  element, 
its  system,  and  its  environment  is  determined  by  what  the 
question  under  investigation  concerns  and  whether  it  is  of  the 
status,  prediction,  or  diagnostic  type.  But,  in  any  event,  one 
thing  is  clear:  the  complete  picture  of  a C&C  element's 

capability  cannot  generally  be  obtained  from  any  small  set  of 
measures.  If  one  wishes  to  know  about  the  C&C  element  - and  if 
system  performance  and  effectiveness  is  of  concern,  then  the  C&C 
element  is  a central  consideration  - then  one  must  be  prepared 
to  deal  with  a large  set  of  numbers  and  to  perform  some  complex 
analyses.  To  be  discussed  in  this  section  are  some  of  the 
reasons  why  this  is  so,  what  is  needed  to  obtain  a complete 
picture,  and  what  is  said  about  C&C  evaluation  in  other  sections 
of  this  report. 

WHY  THIS  IS  SO  The  reasons  why  a large  set  of  measures  are 
needed  in  order  to  completely  analyze  and  evaluate  the  C&C 
element  are  threefold:  (1)  C&C  has  several  kinds  of  relationships 

with  the  system;  (2)  the  ultimate  criterion  is  C&C  effectiveness; 
this  is  reflected  in  system  achievement,  but  system  achievement 
is  also  a function  of  many  other  things;  and  (3)  if  the  question 
is  of  a predictive  or  diagnostic  sort  then  one  must  also  obtain 
meact  *es  of  the  performance  and  design  of  the  C&C  element  itself. 

The  several  relationships  that  the  £&C  element,  the  operator 
and  equipment  components  of  the  plant,  and  the  overall  system  can 
have  with  each  other  have  already  been  discussed  in  pages  15-17, 
terms  of  C&C  definitions  and  models,  and  on  pages  52  - 59,  in 
.erms  of  measures  and  measurement  considerations.  Suffice  it  to 
reiterate  here  that  there  are  limiting  and  enabling  relationships, 
as  well  as  determining  ones,  and  that  time  is  a variable  in  many 
of  these.  The  impact  of  having  these  several  relationships  is 
that  measures,  often  more  than  one,  are  needed  on  each  relation- 
ship - and  that  this  can  lead  to  a large  set  of  measures. 

System  achievement  is  the  responsibility  of  the  C&C  element, 
so  measures  of  system  achievement  with  respect  ot  any  of  its 
objectives  is  also  a measure  of  C&C  effectiveness.  But  system 
achievement  is  a composite  measure  and  it  is  a function  of  several 
things.  All  of  the  multiple  items  making  up  the  composite 
measure,  plus  the  items  exerting  a determining  influence  cn  each 
of  the  multiple  items  are,  in  one  way  or  another,  the  responsi- 
bility of  the  C&C  element  - but  they  are  not  all  directly 
controlled  by  the  C&C  element.  The  determining  item3  include, 
for  example,  mission  environments  (e.g.,  sea  states  and  weather), 
mission  scenario  evolution  (affected  by  such  situational  events 
as  emergencies,  enemy  tactics,  etc.),  the  performances  and  states 
of  each  of  the  system's  components  (e.g.,  equipment  conditions, 
motivation  anu  ability  levels,  performance  characteristics), 
certain  online  actions  of  the  C&C  element  itself,  and  system 
interfaces  with  other  systems.  In  other  words,  wh.i  le  it  is  true 
that  measures  of  system  achievement  are  measures  of  C&C 
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effectiveness#  they  are  also  measures  of  the  effects  of  other 
things  and  are  multiply  determined:  some  of  these  things  are  not 

under  the  control  of  the  C&C  element  at  all  (e.g.,  sea  states)# 
while  others  may  npt  be  effectively  controlled  due  to  limitations 
beyond  the  jurisdiction  of  the  C&C  element  (e.g.,  commands 
received  from  a point  higher  in  the  chain-of- command,  resource 
constraints) . 

The  implication  of  all  of  this  is  that  a large  number  of 
measures  of  the  system  and  its  environment  are  also  needed  in 
order  to  completely  evaluate  C&C  effectiveness.  Data  needs  to 
be  collected  on  many  measures  in  order  to  get  information  on# 
first  of  all,  all  of  those  aspects  of  the  system  for  which  C&C 
is  responsible  and#  second  of  all#  on  those  other  things  which 
are  also  contributing  to  system  achievement  but  may  be  beyond 
the  control  of  the  C&C  element.  It  is  only  thus  that  one  can 
get  information  on  the  several  facets  of  C&C  effectiveness  and# 
also#  on  those  things  which  can  also  vary  the  composite  achieve- 
ment measure  but  which  are  not  "the  fault  of"  C&C  action.  A 
much  more  concise  way  of  saying  all  of  the  foregoing  is  to 
simply  note  that#  for  evaluation  of  C&C  effectiveness,  multiple 
measures  are  needed  of  system  achievement,  as  well  as  measures 
of  the  composite  and  of  some  determining  variables  (cf,  pp. 

- 55).  And  to  underline  again  that  the  C&C  element  is  an 
integral  part  of  the  system,  inseparable  from  it  when  considering 
Cf C effectiveness  questions. 

All  of  the  foregoing  has  had  to  do  with  only  one  aspect  of 
C&C#  that  is,  the  status  of  its  effectiveness.  There  are  at 
least  two  other  aspects  which  also  need  to  be  considered  and 
measured,  however,  if  one  wishes  the  more  complete  understanding 
of  C&C  needed  for  effectiveness  prediction  or  diagnosis  purposes 
(e.g.,  an  Operational  Readiness  Evaluation  f ORE ) ) . These  other 
aspects  are  C&C  performance  and  C&C  design  and  functioning.  If 
one  wishes  to  know  how  effective  a C&C  element  would  be  under  a 
set  of  circumstances  different  from  those  which  have  already 
been  assessed  for  status  purposes  - or  if  the  C&C  element  is 
found  to  be  less  effective  than  desired  - then  one  must  consider 
that  which  determines  C&C  effectiveness?  that  is,  the  perform- 
ance of  C&C  and,  ultimately,  the  design  and  functioning  aspects 
of  C&C.  What  this  means  is  that  the  set  of  measures  must  be 
further  expanded;  for  predictive  and  diagnostic  questions 
measures  are  needed  not  only  of  the  system,  its  environment,  and 
its  plant  components,  but  also  of  the  C&C  element  itself. 

And  the  kinds  of  performance  and  the  aspects  of  design  found  in. 
C&C  are  such  t-.hat  many  measures  are  generally  needed  to  gain 
sufficient  information. 


WHAT  IS  NEEDED  TO  GAIN  A COMPLETE  PICTURE  What  all  needs  to  be 
measured  with  regard  to  the  syp\em  has  been  spelled  out  in  general 
terms  in  the  above  paragraphs . What  needs  to  be  measured  on  the 
C&C  element  itself  is  discussed  in  detail  on  pages  15  and  16 
where  a definition  of  C&C  is  presented.  For  reader  convenience, 
some  facets  of  C&C  effectiveness,  performance,  and  design  and 
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functioning  are  summarized  in  Table  4. 

Unless,  however,  a great  deal  more  is  understood  about  the 
system  than  is  usually  the  case,  its  CSC  element  cannot  be 
evaluated  through  a knowledge  of  just  the  one  system  by  itself. 
Unless  one  understands  the  system  and  its  C&C  element  sufficiently 
well  to  define  a set  of  measures,  all  of  which  are  defined  by 
ratio*  scales  and  have  associated  with  them  a standard,  then  one 
needs  to  be  able  to  compare  the  system  to  other  systems  so  as  to 
gain  at  least  ordinal**  data  on  the  C&C  element.  The  kinds  of 
comparisons  between  systems  that  may  be  useful  in  attempting  to 
reach  an  ordinal  judgment  regarding  the  effectiveness  of  a 
particular  operational  C&C  element  include: 


What  was  achieved  in  the  system  under  evaluation? 


vs. 

What  was  achieved  in  other  similar  system  situations? 


vs . 

What  could  the  C&C  element  have  possibly  achieved  with 
this  system,  given  the  resources  available  over  a 
substantial  preceding  time  period  and  the  operational 
environment? 

Another  way  of  saying  the  foregoing  is  to  note  that  the 
performance  of  an  analysis  is  one  thing  and  is  needed  to  arrive 
at  answers  to  each  of  the  above  questions  individually.  An 
evaluation,  however,  of  how  good  or  bad  a C&C  element  is  requires 
a comparison  of  that  element  against  some  standards  of  achieve- 
ment. Since  we  generally  do  not  presently  have  specific 
standards  of  achievement,  performance,  or  design  for  C&C 
elements  - and,  as  discussed  in  the  foregoing  section,  standards 
existing  for  the  system  overall,  if  any,  are  not  appropriate  in 
and  of  themselves  - an  alternative  is  a comparison  of  the  system 
and  its  C&C  element  against  other  similar  systems  and  against 
analytic  judgments  of  what  should  be  possible,  it  should  be 
quickly  noted  that  if  analytic  judgments  of  the  possible  differ 
very  much  from  what  is  being  achieved  in  other  similar  systems 
then  either  the  analysis  has  been  incomplete  or  in  error,  or  there 
is  a widespread  C&C  problem  {if  the  judgment  of  what  should  be 


♦that  is,  the  distances  between  scale  points  is  known  and  the 
scale  originates  from  a true  zero  point.  If  standards  have  been 
defined  then  data  on  such  scales  will  provide  sufficient  in- 
formation for  evaluation  in  and  of  themselves. 

♦♦ordinal  scales  of  measurement  provide  "greater  than"  and  "less 
than"  relationships  information. 
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TABLE  4.  A LISTING  OF  ITEMS  CONSTITUTING  C&C  EFFECTIVENESS, 
PERFORMANCE,  AND  DESIGN  AND  FUNCTIONING 


C&C  EFFECTIVENESS 

C&C  PERFORMANCE 

C&C  DESIGN  AND 
FUNCTIONING 

System  Effectiveness 

System  Performance 

System  Reliability 

System  Survivabil- 
ity/Vulnerability 

System  States 

(e.g.,  organiza- 
tional and  psycho- 
logical climates, 
component  capabil- 
ity and  motivation 
levels) 

States  the  system 
objectives,  roles, 
general  procedures, 
priorities,  and 
constraints . 

The  general  style, 
manner,  and,  espe- 
cially, the  timing 
of  implementation 
of  the  above  and 
online  mission 
decision-making . 

Utilizes  resources 
to  maintain/en- 
hance the  plant, 
system  interfaces, 
system  environ- 
ment, and  C&C 
capability. 

Institutes  changes 
in  the  foregoing 
so  as  to  modify 
system  parameters 
as  needed  or 
desirable . 

Performs  planning 
activities  to 
decide  future 
system  goals  and 
states,  and  to 
identify/evaluate 
the  requirements 
for  achieving 
these  future  goals, 
goal  changes,  and 
states . 

C&C  element  organiza- 
tional structure 
and  the  associated 
authority /responsi- 
bility breakouts 
vs.  the  definitions 
of  structure  and 
operation  possible 
for  the  system.* 

Design,  management, 
and  use  of  support- 
ing information  and 
data  processing 
systems  by  nodes 
in  the  C&C  chain- 
of-command. 

♦Definitions  of  C&C  structure  and  of  the  supporting  information 
and  data  processing  systems  must  be  in  terms  of  system  objectives. 
It  is  often  the  case  that  the  system  has  two  or  more  objectives 
and  that  each  of  these  can  be  best  achieved  by  a unique  C&C 
structure;  that  is,  that  two  or  more  C&C  structures,  or  an  inte- 
grated one,  m < ' be  needed  in  a system  if  each  of  i«_s  goals  is  to 
be  effectively  achieved. 
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exceeds  reality) , or  systems  are  achieving  on  the  basis  of 
superhuman  effort  (if  reality  exceeds  judgment) . The  latter  two 
cases  are  certainly  grounds  for  action;  in  the  one  case  the 
systems  are  not  performing  up  to  their  capability,  in  the  other 
case  the  systems  are  likely  to  fail  at  any  given  moment  or  under 
any  further  load;  in  either  case,  the  capabilities  of  the 
systems  need  to  be  adjusted  to  an  acceptable  level  of  operational 
capability.  (See  Ref.  7,  p.  28  for  a discussion  of  overtaxed 
operational  systems . ) 

WHAT  HAS  BEEN  SAID  ELSEWHERE  ABOUT  C&C  ANALYSIS  AND  EVALUATION 
Although  this  entire  report  is  a methodology  for  C&C  analysis  and 
evaluation,  only  certain  sections  deal  specifically  with  C&C 
itself.  The  reason  for  this  is  that  an  integral  and  essential 
part  of  any  C&C  analysis  and  evaluation  is  an  adequate  and 
appropriate  analysis  and  evaluation  of  the  system  and  its  plant  - 
manned  systems  analysis  has  therefore  been  a central  concern. 

The  goal  of  this  section  is  simply  to  summarize  the  other  sections 
in  the  report  dealing  more  specifically  with  C&C  analysis  and 
evaluation  per  se. 

C&C  DEFINITIONS  (PAGES  15  AND  18)  It  is  very  difficult,  if 

not  impossible,  to  specify  measures,  analyze  data,  or  evaluate 
results  on  something  which  is  not  clearly  and  explicitely 
defined.  C&C  is  neither  a physical,  a simple,  nor  a stationary 
entity  and,  consequently,  has  often  been  assumed  to  be  undefinable 
and,  therefore,  ignored  in  most  analytic  studies.  Under  the 
assumption  that  we  could  proceed  only  if  we  defined  more  clearly 
what  C&C  is,  and  what  it  is  not,  considerable  effort  was  devoted 
to  the  development  of  C&C  definitions  in  this  program. 

THE  USE  OF  OPERATOR  MODELS  (PAGES  35  - 43)  An  evaluation 

of  alter na'tive  C&C  management  strategies  and  tactics  is  based  on 
an  analysis  to  determine  the  ultimate  effects  these  strategies 
have  on  system  effectiveness  and  performance.  The  use  of  opera- 
tor models  to  evaluate  the  immediate  effects  of  these  alternative 
strategies,  say  on  operator  performance  levels  or  task  errors  and 
omissions,  and  the  consequence  effects  on  the  system  is  discussed. 

THE  C&C  ANALYSIS  MODEL  AND  SYSTEM  RELATIONSHIPS  FORMULATION  (PP. 

60  - 68  ) The  problem  of  modeling  systems  in  a manner  which 

will  permit  a more  realistic  incorporation  of  both  human  and 
equipment  components  and,  further,  would  facilitate  simulation 
of  those  effects  on  the  system  that  the  C&C  element  is  likely  to 
have  was  tackled  in  this  program.  A solution  was  found  through 
the  development  of  simulation  language  programming  techniques 
for  tying  together  inputs  from  the  earlier  stages  of  analysis 
into  a system  model. 

C&C  ANALYSIS,  EVALUATION,  AND  THE  DECISION  TOOLS  AND  UTILITY 
NOTIONS  (PAGES  81  - 86  ) As  already  noted,  the  complete 

analysis  and  evaluation  of  a C&C  element  is  a complex,  lengthy, 
and,  therefore,  expensive  process.  Because  time  and  resources 
are  and  always  will  be  limited  and  competed  for,  the  analyst  must 
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carefully  consider  what  it  is  he  really  wants  to  know  and  just 
what  that  information  is  worth.  The  application  of  decision 
and  utility  concepts  to  an  especially  pressing  problem,  system 
operational  readiness,  is  discussed. 
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VI.  ANALYSIS  PROCESS  EVALUATION: 

APPLICATION  OF  DECISION  CONCEPTS 


DATA  INTERROGATION 

USING  THE  DATA  BASE  Assuming  that  the  analyst  and  the  decision 
maker  have  a large  quantity  of  data  available,  the  question  is: 
how  to  use  the  data  base?  A passive  response  of  using  only 
(and  all  of)  the  data  supplied  will  result  in  nothing  but 
confusion.  What  is  needed  for  the  analyst  is  an  active  strategy 
of  exploring  the  data  base. 

At  least  two  such  strategies  may  be  distinguished: 
sequential  data  interrogation  and  sequential  question  interroga- 
tion. These  will  be  described  in  the  following  paragraphs. 

In  both  cases,  the  use  of  the  term  "sequential”  implies  that 
a single  pass  will  not  be  adequate.  It  is  very  doubtful  that 
answers  to  significant  questions  will  be  derived  on  a single 
request.  Indeed,  given  the  state  of  some  current  data  bases, 
even  very  "simple"  questions  cannot  be  answered  immediately. 

For  example,  it  is  apparently  not  easy  to  get  the  answer  to  the 
following  question:  "On  this  day  how  many  people  are  serving  on 

active  duty  in  the  U.S.  Navy?"  The  analyst,  therefore,  should 
expect  an  iterative  search  procedure. 

SEQUENTIAL  DATA  INTERROGATION  So  much  raw  data  are  now  avail- 
able to  the  analyst  that  he  may  be  tempted  to  survey  the  data 
base  looking  for  questions.  This  technique  is  effectively: 

Given  uhe  answer  what  is  the  question?  At  best,  this  is  a 
formalization  of  searching  for  serendipity.  While  the  analyst 
will  possibly  uncover  accidental  discoveries  it  is  not  probable 
they  will  be  either  desirable  or  useful.  One  should  hasten  to 
add  it  is  not  impossible.  But  it  is  not  efficient. 

Using  the  technique  of  sequential  data  interrogation,  the 
analyst  faces  at  least  two  problems:  time  and  inadequate 

information.  Manual  data  interrogation  is  an  extremely  slow 
process,  and,  while  it  may  have  some  emotional  rewards,  will 
probably  create  an  answer  long  after  the  answer  is  required. 

A recen^  innovation  for  scanning  data  bases  has  been  the 
development  of  semi-automatic  decision  aiding  techniques.  But 
these  methods  assume  that  the  analyst  is  asking  some  generic 
type  of  question  (e.g.,  the  links  between  kinds  of  data). 

Inadequate  information  means  that  raw  data  are  rarely  in  a 
form  which  directly  answers  questions.  Forms  of  transformation 
on  the  data  will  very  frequently  be  required,  unless  the  question 
requires  an  elementary  counting  procedure  or  a known,  convention- 
al, algorithm.  The  most  reliable  computer  information  systems, 
such  as  for  accounting  or  statistical  analysis,  are  precisely  of 
this  kind.  But  even  these  systems  assume  a fixed  set  of 
"questions"  for  counting  such  as  net  profit  or  statistical 
significance  of  a variable. 


SEQUENTIAL  QUESTION  INTERROGATING  This  entire  report  has 
stressed  the  point  that  the  analyst  should  approach  the  C&C 
elrment  and  its  system  in  a question  mode.  A previous  section 
(pp.  48  - 50)  has  stated  three  general  classes  of  questions: 

STATUS:  How  goes  it? 

PREDICTION:  How  will  it  go? 

DIAGNOSIS:  What  is  going  wrong? 

Sequential  question  interrogation  demands  that  the  analyst  form 
his  questions  as  precisely  as  possible  before  entering  the  data 
base.  In  short,  the  analyst  should  ask  himself  as  clearly  as 
possible:  What  do  I want  to  know?  But,  it  is  rarely  possible 

to  state  meaningfully  any  significant  question  before  examining 
the  data.  Therefore,  he  must  be  prepared  to  revise  and  restate 
his  question  in  light  of  the  data.  The  analyst  might  keep  in 
mind  two  methodological  questions  - 

What  Do  You  Mean?  and 
How  Do  You  Know? 

when  he  is  employing  the  sequential  question  interrogation 
technique. 

A MINIMIZATION  AXIOM  It  should  be  understood  that  advancing 
from  status  to  prediction  to  diagnosis  questions  will  have  two 
adverse  consequences  on  data  demands:  (1)  there  will  be  a marked 

increase  in  data  requirements  and  (2)  there  will  be  a significant 
increase  in  analytic  (transformation)  steps. 

Therefore,  the  analyst  should  attempt  to  minimize  the 
information  he  demands  from  the  C&C  element  and  its  system, 
consistent  with  the  questions  he  is  asking.  Diagnostic  questions 
should  never  be  asked  unless  the  answer  to  a status  question 
reveals  a clear  and  significant  deficiency. 

For  system  control  and  planning,  the  most  significant  type 
of  question  the  analyst  and  the  decision  maker  will  ask  concerns 
system  prediction.  It  is  in  the  nature  of  the  prediction  of  most 
process  events  that  the  longer  ahead  one  wishes  to  predict  the 
less  valid  and  reliable  the  prediction.*  The  analyst  should  ask 
whether  or  not  five  and  ten  year  prediction  requests,  for  example, 
are  meaningful.  The  minimization  axiom  applies  to  the  time 
duration  of  prediction. 


*A  fact  which  makes  one  envy  the  predictability  of  astronomical 
events.  This  success,  however,  is  based  on  (1)  good  quantita- 
tive theory,  (2)  over  3,000  years  of  data  collection,  and  (3)  a 
reasonably  stable  process.  These  conditions  do  not  prevail  for 
manned  systems. 


The  minimization  axiom  also  applies  to  the  depth  and 
precision  of  prediction  questions.  Answers  can  be,  and  are, 
invented  for  detailed  prediction  questions  of  time-distant 
events,  but  their  validity  and  usefulness  are  doubtful.  Indeed, 
they  may  be  harmful  in  that  they  may  well  be  incorrect.  For 
example,  five  and  ten  year  facilities  predictions  for  a system 
are  of  value  only  in  so  far  as  the  predicted  functions  of  the 
system  are  reasonably  defined.  Facilities  have  been  created  for 
systems  which,  in  the  meantime,  have  disappeared. 

The  minimization  axiom  seeks  to  reduce  data  demands  on  a 
system  and,  at  the  same  time,  to  structure  techniques  for 
meaningful  data  search.  Vague  and  continual  system  interrogation 
is  a costly  process,  and  can  result  in  degradation  of  system 
performance.  Time  spent  collecting  useless  data  is  time  not 
spent  in  performing  system  functions. 

SYSTEM  OPERATIONAL  READINESS 

AN  EXAMPLE  Many  of  the  problems  raised  in  this  report  can  be 
exemplified  in  a very  common  question:  What  is  the  operational 

readiness  of  the  command  and  control  system?  Some  of  the 
difficulties  encountered  in  answering  this  question  may  be 
illuminating . 

THE  QUESTION  First  of  all,  the  general  question  should  be 
restated  as  follows: 

STATUS:  Can  the  overall  system  meet  the  present 

threat? 

PREDICTION-:  Will  the  system  meet  future  threats? 

DIAGNOSIS:  If  not,  why  not? 

It  is  to  be  noted  that  the  questions,  as  re-stated,  refer  to 
the  ability  of  the  total  system  to  respond  to  an  external  threat. 
This  is  consistent  with  definitions  of  the  functions  of  the  CSC 
component  with  regard  to  total  system  performance  (see  pp.  15  - 
16) • Questions  about  the  C&C  component  are  not  of  most  immediate 
and  general  interest;  the  internal  performance  of  the  C&C 
component  is  only  a part  of  total  system  performance,  contributing 
to  it  as  do  the  other  system  components.  It  is  only  if  the 
answers  gained  from  total  system  status  questions  provide 
insufficient  information  or  are  not  acceptable,  that  one  proceeds 
predictive  and  diagnostic  question  forms;  and  it  is  only  here 
that  one  begins  to  measure  and  evaluate  such  system  components 
as  the  C&C  element. 

THE  MODEL  Quantitative  models  for  the  above  questions  have 
generally  been  stated  in  the  form  of  a linear  multiple  regression 
equation: 
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That  is,  some  set  of  system  component  (predictor)  variables  are 
used  to  predict  system  performance  (criterion  variables) . 

In  fact,  this  model  is  of  most  use  with  respect  to  diagnos- 
tic and  tradeoff  prediction*  questions.  If  the  criterion 
variable  value  is  unacceptable,  then  the  causes  and  basis  for 
change  are  presumably  to  be  found  somewhere  within  the  predictor 
variables.  For  status  and  straight  prediction  problems,  only 
the  direct  expression  of  operational  readiness  (a  single  criter- 
ion variable)  is  normally  desired. 

THE  SINGLE  CRITERION  Most  users  of  information  desire  a simple 
expression  of  readiness.  Examples  are  (1)  availability  of  the 
system,  (2)  percent  capability,  or  (3)  amount  committed.  All  of 
these  are  composite  criteria.  They  provide  a quick  and  easy  to 
understand  measure  of  system  operation? 1 readiness. 

However,  they  may  not  be  meaningful.  To  say  that  a system 
is  50%  operationally  ready  is  insufficient  to  answer  the  status 
question.  That  value  is  neither  good  nor  bad  per  se.  The 
question  remains:  Is  50%  sufficient  to  meet  the  present  threat? 

If  the  answer  is  "no",  then  diagnostic  interrogation  will  be 
necessary. 

MINIMIZATION  AXIOM  APPLIED  A common  mistake  is  to  assume  that 
any  "low"  composite  criterion  value  (50%,  75%,  etc.)  will  mean 
an  unacceptable  system  state.  This  may  not  only  be  wrong,  it 
may  also  lead  to  unnecessary  data  demands. 

What  must  be  done  is  to  evaluate  the  composite  criterion. 

It  would  be  desirable  to  set  a mini-max  threshold  of  acceptabil- 
ity. This  would  add  the  binary  judgment  of  "acceptable- 
unacceptable"  to  the  composite  criterion.  If  acceptable,  no 
further  data  interrogation  is  required.  If  not  acceptable,  then 
further  investigation  is  obviously  warranted. 

The  minimization  axiom  calls  for  a minimum  call  on  data 
demands.  It  seems  particularly  appropriate  in  this  specific  case. 


*If  one  wishes  to  perform  tradeoff  estimations  of  prediction 
processes,  then  the  predictor  variables  must  be  identified  as 
they  are  for  diagnostic  questions  so  that  alternative  values 
on  these  parameters  can  be  manipulated. 
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But,  to  be  applied,  external  threshold  criteria  must  be  developed; 
some  technology  is  available  from  computer-based  management 
information  systems. 

MODELLING  PROBLEMS  Assessment  and  evaluation  require  modelling. 
In  the  case  of  systems  several  problems  can  be  noted: 

1.  Multiple  criterion  variable  sets  are  probably  necessary 
for  questions  other  than  basic  status  questions..  As  an  example, 
in  C&C,  in  addition  to  some  "avail ability”  measure  one  might 
also  wish  to  assess  "responsiveness".  A system  can  be  available 
for  use,  yet  be  incapable  of  quick  and  competent  response. 

2.  If  a linear  multiple  regression  model  is  used,  then  the 
question  may  be  raised  if  that  is  the  appropriate  form.  It  is 
doubtful  that  predictor  variables  in  a command  system  combine  in 
an  additive  fashion.  But,  as  noted  before  (Chapter  V)  the 
simplest,  most  reasonable,  model  is  the  best  choice.  And, 
"reasonable"  is  measured  by  the  degree  of  acceptability  that  the 
model  provides.  Answers  to  the  fifth  decimal  are  not  cost- 
effective  when  the  whole  number  provides  an  acceptable  answer 
for  decision. 

3.  Assignments  of  weightings  to  the  predictor  variables 
(and  criterion  variables  if  a multiple  criterion  is  used)  has 
been  a very  difficult  technical  problem.  These  can  be  generated 
either  through  (1)  analysis,  (2)  expert  opinion  or  (3)  empirical 
data  (Ref.  22) . As  mentioned  before  (pp.  69-74)  it  may  not  be 
necessary  for  precision  at  the  level  previously  assumed. 

BENEFIT  AND  COST 

VALUE  OF  INCREASED  INFORMATION  No  one  knows  for  sure  what  impact 
increased  data  and  information  have  had  on  C&C  and  management 
information  systems  (Ref.  5) . The  impact  just  has  not  been 
subjected  to  empirical  test. 

This  report  has  been  concerned  with  the  potential  negative 
impacts  of  demanding  too  much  data  and  improper  generation  and 
utilization  of  data.  On  the  positive  side,  it  has  emphasized 
systemati  strategies  toward  collecting  information. 

But,  the  objective  demonstration  of  the  benefits  of  more 
information  or  C&C  remains  to  be  performed. 

BENEFIT  ANALYSIS  We  should  like  to  be  able  to  determine  the 
cost-effectiveness  of  data  collection  in  C&C.  Unfortunately, 
past  attempts  have  stressed  cost  without  adequate  consideration 
of  effectiveness  (benefit) . 

One  approach  (Ref.  4),  however,  has  structured  a model  for 
benefit  assessment.  Three  vectors  are  established: 


Realized  Potential  Received  Utilization 

Value  ■ Contribution  X Value  X Value 
(Benefit)  (P)  (R)  (U) 

stressing  the  value  of  the  information  based  on  specifications 
(P) / user  receipt  of  the  information  (R) , and  receiver  utiliza- 
tion of  the  information  (U) . 

A number  of  transformations  are  possible  within  this  model 
but  two  seem  particularly  important: 

1.  It  is  desirable  to  compare  what  Lr  obtained  relative  to 
what  could  be  obtained.  This  is  termed  the  " realization/potential 
ratio"  and  is  expressed  by: 


Realized  benefit 
Potential  contribution 


Index  of  Potential  Realized 


2.  Fundamental  are  the  perceptions  of  the  users;  this  is 
expressed  by: 


User's  perception  of  realized  oenefit 
Producer's  perception  of  realized  benefit 


Congruence  Index 


Emphasis  here  is  placed  on  the  continual  dissonance  between 
producers  and  consumers.  And  it  places  a focus  on  systematic 
evaluation  of  user's  perceptions.  In  the  final  analysis,  the 
benefit  of  any  system  rests  ultimately  upon  that  perception. 
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